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Executive Summary 



Over the past five and one-half years, the Idaho Educational Technology Initiative has 
distributed 56 million dollars to Idaho public schools. This report reviews the progress made 
toward accomplishing goals established by the Idaho Council on Technology in Learning. It is 
based on data collected during visits to 36 schools in 16 school districts, and the review of 48 
technology research and evaluation studies conducted in Idaho schools. From these visits and 
document review, I have concluded: 

• There is convincing evidence that the Idaho Educational Technology Initiative has been 
implemented in accord with ICTL Goals. At the present time, Goal # 1, Technology 
Integration, is being given highest priority. Idaho educators at all grade levels have turned 
their attention from establishing a technological infrastructure to integrating technology into 
daily classroom instruction. 

• Recent research and evaluation studies document ways in which technology can be used to 
enhance student learning. These studies have been conducted by State Department of 
Education staff. State Division of Vocational Education staff, college of education staff, and 
teachers. 

• Three research studies deserve special attention because of innovative methodology (The 
Idaho Technology Initiative: An Accountability Report to the Idaho Legislature by Cliff 
Green and Mike Rush) , compelling findings (Mathematics Technology Project by Albert W. 
Strickland and Jack A. Coffland), or comprehensive scope (Computers in the Classroom: 
The Impact of Technology on Student Learning by Dawn Stram Stratham and Clark R. 
Torell). 

• Idaho educators have made less progress toward Goal 6 (Evaluation) and Goal 8 (Systems 
Support) than toward the other ICTL goals. Accomplishing both these goals demands 
expertise not often found in school districts and requires expenditures of time and money 
not anticipated by districts. If statewide information is desired on the implementation and 
impact of educational technology, it will probably be necessary for the State of Idaho to define 
a statewide evaluation strategy and fund data collection and analysis. Similarly, consideration 
of the State role in helping districts provide technology support to teachers, students, and 
administrators is warranted. Regional state-funded network support centers could provide 
important assistance to many Idaho school districts. 

• A variety of strategies are being used by Idaho colleges of education to support the Idaho 
Educational Technology Initiative, and these strategies are effective. Teachers consistently 
report they are receiving quality training from BSU, ISU, University of Idaho, and LCSC. The 
challenge the colleges face, however, is to offer training to all teachers that need it. There 
are approximately 15,000 teachers in the State of Idaho, and a training staff at the colleges 
of education of approximately 20. It is clear that these few individuals cannot directly meet 
the diverse technology needs of teachers in geographically dispersed districts. Idaho colleges 
face the task of maintaining technology training programs that produce teachers who can 
train and support other teachers within the same building and district. 
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• Technology is only a tool for teaching and learning, and future research and evaluation 
studies should consider the context in which this tool is used. Special attention should be 
given to the model of curriculum and instruction being used in concert with the technology 
and the characteristics of the students, classes and schools participating in the research or 
evaluation. 

• Future statewide evaluation efforts should be defined in advance of program implementation, 
be conducted by a disinterested external evaluator, focus on explicitly defined program 
goals, consider technology use in the context of subject area and curriculum/instructional 
approach, and, wherever possible, use existing data collection methods. 

• The “Accountability” data set collected by Cliff Green and Mike Rush should be subjected to 
further statistical analysis. 
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Preface 



The purpose of this report is to provide an outsider s look at progress made implementing 
the Idaho Educational Technology Initiative. It consists of the following chapters: 

Chapter 1: Funding the Idaho Educational Technology Initiative chronicles the 
expenditures made from the start of the Initiative. 

Chapter 2: A Review of Research and Evaluation Studies Focusing on Technology 
Integration in Idaho Schools assesses the quality and reports the findings of recent 
technology research conducted in Idaho. 

Chapter 3: ICTL Goals and Experience of Idaho Educators: A Report on Site Visits to 16 
Idaho School Districts discusses my observations of how Idaho educators are implementing 
the Technology Initiative’s goals. 

Chapter 4: Assessing the Impact of Educational Technology: Recommendations for Future 
Evaluations describes two evaluation models that could provide useful information to both 
educators and policy-makers. 

I hope that the information contained in this document will be useful to Idaho Legislators and 
educators. 

Many individuals have made contributions to this document, and I would like to 
acknowledge their contributions. A number of people shared their perspectives on the Idaho 
Educational Technology Initiative. I appreciate the time and information they have given me. 
They include Senator Melvin Richardson; Dr. Robert C. West, Chief Deputy Superintendent of 
Public Instruction, Idaho Department of Education; Rich Mincer, David Breithaupt and the staff 
at the Bureau of Technology Services, Idaho Department of Education; Nancy Zofran, Chief 
Technology Officer, State Board of Education; John Davis, Scott Coleman, Carolyn Thorsen, Al 
Strickland, Heidi Rogers, Eddie Kennedy and the staff who support the Regional Technology 
Advisors. 

I am indebted to the many principals and teachers who have invited me into their schools 
and answered my questions. Sixteen District Superintendents and Technology Advisors have 
introduced me to their districts and showed me the equipmeni; they have purchased with funds 
from the Technology Initiative. A variety of school district staff have been extremely helpful, 
from receptionists responding to my requests for directions to Deputy Superintendents 
accompanying me during a visit. I am deeply appreciative of the many Idaho educators who gave 
of their time and themselves in the course of assembling this report. 
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Finally, I am indebted to Marie Kanarr and Kevin Moriarty at the Buck Institute for 
Education. These individuals reviewed initial drafts and helped produce the report. To all, a 
hearty Thank You! 

John R. Mergendoller, Ph.D. 

Novato, CA 

December 1999 
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Chapter 1 : Funding the Idaho 
Educational Technology Initiative 

A brief history of the legislation with yearly appropriations 

Introduction to the Legislation 

In 1994, the Idaho Legislature passed HB901: the Idaho Educational Technology Initia- 
tive. According the State Board of Education and the initiative text, the PURPOSE of HB 901 
was: 



• To establish a mechanism to assure the future coordination and effective implementation 
of state-funded learning technologies that have been handled piecemeal and haphazardly 
in the past. 

The initiative text listed the following as primary GOALS: 

• Promote the effective use of learning technologies; 

• Meet urgent needs for equipment, software and training; 

• Coordinate delivery of related services; 

• Maximize the benefits of these technologies; and 

• Minimize waste and duplication. 

In order to accomplish these goals, the legislation provided for the following ADMINIS- 
TRATION: 



• Idaho Council of Technology in Learning (ICTL) under the State Board of Education; 

• The appointment of members to the council; 

• The establishment of the responsibilities of the council; and 

• A public school technology grants program administered by the council subject to the 
approval of the State Board of Education. 

The Idaho Legislative Services Office described the legislation’s major OBJECTIVES. 

® To provide all Idahoans access to an evolving mix of learning technologies that involve 
communications by voice, video and with written data; 

• To maximize the benefit from dollars spent on learning technology in higher education, 
vocational education, Idaho libraries and the public schools; and 

• To adapt and fully utilize telecommunications and information technology to help meet 
the states constitutional requirement to provide a “thorough system of education'' for 
Idaho students; 
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The same Legislative Services Office report explained that the legislative objectives will 
be met through the following STRATEGIES: 

• Coordinate telecommunications and information technology activities; 

• Train current and future teachers to use learning technologies; 

• Fund necessary equipment, technical support and communication links to provide ample 
access to learning technologies; and 

• Ensure that equipment, technical support and communications links for the institutions 
and agencies are provided in a consistent and efficient manner so their employees, patrons 
and students may interact electronically with each other and the international information 
highway. 

Expenditures for 1994-1999 

As a result of this initiative, over 56 million dollars have been spent on public schools in 
the state of Idaho in the past five and a half years. Beginning with the allocation of 10.4 million 
dollars in the 1994'95 school year, approximately 10.3 million dollars have been spent on 
educational technology in public schools in every year. This 1999-2000 school year marks the 
sixth year of operation of the Idaho Educational Technology Initiative. 

In the first four years (1994-1995. 1995-1996. 1996-1997. 1997-1998). 10.4 million 
dollars were allocated each year to fund the technology grants created by the Idaho Educational 
Technology Initiative. During the 1998-1999 school year. 10.0 million dollars were spent on 
public school grants. To date. 4.8 million dollars from the 1999-2000 grants have been 
distributed. Figure 1.1 illustrates spending levels for the past six years. 



Figure 1.1: Idaho Educational Technology Initiative 
Public School Grants for the Years 1995-2000 
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Funding Formulas 



Of the $10.4 million distributed in 1994, 70%, was distributed to schools on a non- 
competitive basis. The formula for distribution was as follows: $20,000 went to each of 112 
districts as a basic grant, for a total of $2,240,000. An additional $20.63 per student was 
distributed based on fall 1994 enrollment, for a total of $4,889,400. The remaining 30% of the 
money allocated, $3,120,000, was distributed through competitive grants. 

In the 1995-96 school year, the money was distributed exclusively through non- 
competitive grants. This formula gave each district $20,000 and $35 per student. In the third 
year of the initiative, the funding formula was changed again. The revised formula, which also 
determines current allocations, allots a $20,000 base for school districts enrolling over 100 
students, a $5,000 base for school districts enrolling under 25 students, and a sliding base for 
school districts enrolling between 25 and 100, calculated by multiplying the average daily 
attendance (ADA) by $200. For example, a school district enrolling 26 students would receive 
a base of $5,200. A school district enrolling with 98 students would receive a base of $19,600. 

After each school district receives its base grant, the remainder of the money is distributed 
proportionally, based on the ADA. Thus, the remainder of the money is divided by the total ADA 
in Idaho and that amount is distributed according to the ADA for each school district. For the 
five completed years that the initiative has been in place, grants have distributed an average of 
$42.40 per student per year, for a total of $212.75 per student over the first five years. 

Expenditure Trends 

Based on budget submissions by individual districts, overall spending for the educational 
technology grants has been calculated for the first three years of the initiative. The purchases 
are grouped into three categories: personnel costs, operating expenditures, and capital outlay. 
Table 1.1 illustrates significant spending trends over the first three years. 



Table 1.1: Idaho Educational Technology Initiative Public School 
Grant Spending Trends for the Years 1995 through 1997 



Public School Grants 


FY 

1995 


%of 

Total 


FY 

1996 


%of 

Total 


FY 

1997 


%of 

Total 


Personnel Costs 


742,689 


9% 


1,257,151 


11% 


1,783,227 


16% 


Operating Expenditures 


1,043,204 


12% 


1,654,894 


15% 


2,060,263 


19% 


Capital Outlay 


6,650,904 


79% 


8,163,041 


74% 


7,109,795 


65% 


Total Funds Spent 


8,436,797 




11,075,086 




10,953,285 




Carryover 


1,919,084 




1,163,998 




1,184,202 





This table demonstrates a trend towards increased spending on operating and personnel, 
and decreased spending on capital outlay. In the first year of the initiative, almost 80% of the 
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funds for the school districts was spent on capital investments. Two years later, in the initiative’s 
third year, only 65% of the funds available was spent on capital investments. As the amount of 
equipment purchased decreases, the amount of money being spent on personnel and operating 
costs has increased equally. In the first three years, the amount of money which went to 
operating expenditures increased from only 1 2% to almost 20% - a fifth of the total expenditure. 
Personnel cost expenditures increased from 9% to 16%. Unfortunately, similar information is 
unavailable for the subsequent three years. 

The funding of the Idaho Educational Technology Initiative represents a compelling 
commitment on behalf of the Idaho Legislature to improve schools and, in combination with 
funding made available through the J.A. and Kathryn Albertson Foundation, sets the stage for 
Idaho to become a leader in the integration of technology into schools, and more importantly, 
into the process of instruction. 
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Chapter 2: A Review of Research and 
Evaluation Studies Focusing on Technology 
Integration in Idaho Schools (ICTL Goal #1) 

Plan for the Chapter 

The following chapter reviews recent research and evaluation studies focusing on 
technology integration in Idaho Schools. The chapter is divided into six sections. 

I first discuss how the research and evaluation studies were collected, and the criteria 
used to determine whether individual studies should be included in my review. (Summaries of 
all research and evaluation studies reviewed appear at the end of this report in Appendix A: 
Research Summaries.) I then describe the dimensions used to evaluate the assembled studies , 
the quality criteria applied to each dimension, and the standard used to define individual studies 
as trustworthy. Finally, I summarize the findings of the “trustworthy” studies, and draw general 
conclusions about the technology evaluation efforts conducted in Idaho to date. 

Assembling Studies for Review 

According to Connections: A Statewide Plan for Technology in Idaho Public Schools, the 
purpose of Technology Integration is “To improve the quality and effectiveness of classroom 
instruction and learning for all learners by integrating technology-based resources in conj unction 
with state curriculum guides. ” ^ There are at least four aspects of Technology Integration that are 
necessary if technology is to “improve the quality and effectiveness of classroom instruction and 
learning.” These are: 1) Technology acquisition; 2) Technology training; 3) Technology use; and 
4) Technology impact. 

To assemble studies with findings relevant to each of these areas, I used a funnel 
approach. I began by seeking out any document that reports on or examines one of these four 
dimensions of technology integration in Idaho schools. A number of different individuals and 
institutions responded to my requests, and I received relevant studies from the Bureau of 
Technology Services, Idaho State Department of Education, from the university-based Regional 
Technology Advisors, and from school district personnel I met during visits to Idaho School 
districts. 
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Criteria for Including Studies in the Review 



Once I had gathered all the documents I could find relevant to the four dimensions of 
Technology Integration, I used the following criteria to determine whether the reports I had 
assembled should be included in the body of research to be reviewed. 

1. The report must contain sufficient information to allow me to judge its quality. 
It must explain clearly the purpose of the research/evaluation, describe how it was 
conducted, depict the research participants and tell what subject area was of 
interest. 

2. The report must contain analyzable data. It must go beyond opinion, stories, and 
anecdotes and contain information susceptible to analysis. 

3. The report must describe completed research/evaluation studies. In-process 
studies with a promise of completion were ignored. 

Out of the 56 documents I assembled, 50 studies met these three inclusion criteria. They vary 
according to methodology used and quality of design and execution, but, at a minimum, they 
provide information about how teachers are using and responding to technology, and the impact 
that technology has had on a specific group of students. As the distinction between research and 
evaluation studies is not a clear one, I have not attempted to separate traditional research from 
applied evaluation research, but considered research and evaluation reports together. 

Dimensions of Study Quality 

Once the group of reports with sufficient information was identified, I reviewed each 
study and made a judgment about whether the results of these studies were credible and could 
be generalized to Idaho education. In my assessments, I have sought to make a fair and 
reasonable judgment essentially similar to that which would be made by other professional 
researchers and evaluators. I have not made allowances for background or training of the 
individual who conducted the research or evaluation. I have focused solely on the 
trustworthiness of the research results and its potential to provide useful information about the 
role and impact of technology in Idaho education.^ 

Assessment Dimensions. After reading each report, I rated each of the assembled studies 
using the following three dimensions: 

• Methodological Rigor. Has the research been described, conceptualized and carried 
out in such a way that one can have confidence in the author’s results and 
conclusions? 

• Generalizability to Idaho Teachers and Students. Can the research results and 
conclusions be applied to Idaho students and teachers other that those on which the 
study was conducted? 
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• Overall Quality of Research/Evaluation. Looking at the research as a whole, does it 
contribute to our understanding of the question addressed? Should its conclusions 
be taken seriously? 

For each of these dimensions I established a rating scale defined by either three or four 
scale points. Studies with higher scores demonstrate more rigorous methodology, greater 
generalizability, and higher overall quality than studies receiving lower scores. Although the 
rating of individual research and evaluation studies is based solely upon my judgment, I have 
applied widely accepted standards for the conduct of educational research and evaluation. 

Assessing Study Quality 

Methodological Rigor. When evaluating a research or evaluation study, it is important 
to consider the methodological rigor with which the study was conducted. A study’s 
methodology includes decisions made about the study design, the procedures used to collect 
data, and the manner in which data were analyzed. In general, findings from studies that reflect 
a careful and well-conceived methodology can be trusted. Findings from studies where the 
methodology is deficient invite conflicting interpretations and unresolvable disputes. I have 
assessed the methodological rigor using the following scale points: 

0 Serious methodological problems invalidate results 

1 Methodological problems compromise trustworthiness of results 

2 Acceptable methodology; trustworthy results 

3 Excellent methodology: compelling results 

Studies rated 0 have one or more significant methodological problems that make it 
impossible to trust the results and conclusions reported by the author. These problems can 
include errors in designing the study, developing the data collection strategy, selecting or 
creating the data collection method, or analyzing the data. Studies rated 0 should be ignored. At 
the other end of the scale, a rating of 3 indicates that exemplary educational research practices, 
as described in educational research methods textbooks, were followed. The results and 
conclusions drawn from such studies should be taken at face value, and should be disseminated 
widely throughout and beyond Idaho. Studies rated 3 are methodological modefs. 

The ratings of 1 and 2 are more difficult to define. Most research conducted in ongoing 
social institutions, like schools, is done under less than optimal research conditions. This reality 
can jeopardize the validity of results. Consider, for example, a research project to examine how 
much students learn in the course of a technology integrated instructional unit in several classes 
taught by the same teacher. Such a research question requires collecting student learning data 
before the class begins the integrated instructional unit. But unexpected changes in school 
schedules and the reality of student absences sabotage plans for data collection, and the teacher 
finds that some students and classes have already begun working on the unit before the pre-tests 
are administered. 
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What are the consequences of this delay? Does it invalidate the study? Or does it simply 
raise an issue the reader needs to think about in evaluating the research results? Studies rated 
1 and 2 fall in the gray area between invalidity and compellingness. Findings and conclusions 
from studies rated 1 and 2 should neither be dismissed, nor should they be uncritically 
embraced. Studies need to be considered individually. In general, the methodology employed in 
research and evaluation studies rated 1 is “good enough” that the reader should consider that 
the findings may be true, but there is no certainty that this is the case. Studies rated 2 follow 
generally accepted methodological approaches, but could have been improved by changes in 
design, instrumentation, or analysis. The findings of these studies should be trusted, although 
it is possible to find minor methodological flaws. 

Generalizability. In addition to the methodological rigor of the research studies, I have 
considered their generalizability to Idaho education. Most often, the generalizability of a study 
depends upon the way the study participants were selected. Random samples drawn from 
representative schools yield results that can be applied to the sorts of schools from which the 
sample was drawn. A study of a single classroom of students in a unique school can not usually 
be generalized beyond the classroom (and teacher) involved in the research because other 
students (and teachers) may be different. 

In evaluating the research summarized in this chapter, I have used the following scale 
points to rate the study’s generalizability: 

0 Results can not be generalized beyond current research/evaluation 

1 Results can be generalized with caution 

2 Results widely generalizable 



Like the unambiguous meaning of the scale points terminating both ends of the method- 
ological rigor rating, the generalizability of studies rated 0 and 2 is unequivocal. Studies rated 
0 should be considered a portrait of the students and teacher(s) on which they were conducted; 
no implications should be drawn regarding whether similar findings would be found with 
different teachers and students. Studies rated 2 employed a sampling strategy that allows findings 
to be generalized to other students and schools. 



Between these extremes, there is again a gray area. It is uncertain whether studies rated 



1 



i will generalize to other groups of j>tudentj> and teachers, and consequently, gene 
must be made cautiously. The reader should carefully consider the factors that would support 
generalization (e.g., a similar situation unaffected by extraneous influences) and those that 
would challenge it (e.g., something unusual or special about the context in which the study was 
conducted.) 



Overall Quality. Taking the methodological rigor and the generalizability of the studies 
reviewed together, I have arrived at a summary rating of the overall quality of the research or 
evaluation study. This is, in common parlance, the bottom line. This scores is not a simple 
average of methodological rigor and generalizability ratings. It takes these ratings into account, 
but is a more global assessment of the study’s quality. The purpose of this rating is to indicate 
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how much trust should be put in this study as an indicator of the processes and consequences 
of technology integration. I have used the following scale to communicate this judgment: 

0 Untrustworthy; conclusions should be ignored 

1 Flawed research/evaluation; consider conclusions skeptically 

2 Competent research/evaluation; conclusions appear sound 

3 Exemplary research/evaluation; conclusions have import for Idaho education 

As with the previous rating scales, the extreme values are unambiguous. A study rated 
0 should be ignored, while a study rated 3 is unusual in the quality with which the research or 
evaluation was conducted. Such work should be given careful study and attention. 

Between these two extremes, lie the bulk of studies reviewed. Those rated 1 are flawed 
in some way. Findings and conclusions should not be ignored, but they should be approached 
with caution and healthy skepticism. Studies rated 2 are well-conceived and conducted; their 
conclusions are based on sound evidence and are worthy of attention. 

Credible Research and Evaluation Studies 

To identify research and evaluation studies that document the processes and outcomes 
of technology integration in a professional and persuasive manner I considered the Overall 
Quality rating I had assigned to each study, and set a rating of 2 as the cutoff point. As noted 
above, studies with a rating of 2 are characterized on the rating scale as "competent 
research/evaluation; conclusions appear sound." Twenty-two of the research and evaluation 
reports I reviewed received an "Overall Quality" rating of 2 or 3. I believe the studies with a 
quality rating of 2 would be convincing to an unbiased, outside observer, and studies rated 3 
would be seen as a compelling research or evaluation study worthy of significant consideration. 
Table 2.1 displays the date, author, title and findings of the "credible" research and evaluation 
studies receiving a rating of 2 or higher. It also indicates the affiliation (e.g.. State Department 
of Education, school district, university, etc.) of the author. 

Reflections on the Credible Studies 

Table 2.1 displays a number of different types of research and evaluation studies. This 
suggests that there are a number of ways classroom teachers, university researchers, and state 
department staff have contributed to the understanding of technology integration in Idaho 
schools. 
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Table 2.1: Credible Research and Evaluation Studies 



Affiliation 
I Date 


Quality 

Rating 


Report # 
Author 


Title 


Results or 
Conclusions 


Idaho Legislative 


2 


(1) 


Interim Evaluation 


The purposes of the legislation have 


Services Office 




Gordon 


of the Idaho 


been achieved and that several of the 


12/26/95 




Fisher 


Education 
Technology 
Initiative of 1994 


goals have been attained or progress 
made toward attainment . . . the 
programs have not been in operation 
long enough to evaluate the relative 










impact, costs and benefits. 


Idaho State 


2 


(2) 


Interim Report of 


The largest single category of 


Department of 




Idaho State 


the Idaho 


hardware being purchased is 


Education 




Department 


Educational 


computers. The student to computer 


1/22/96 




of Education 


Technology 
Initiative of 1994 
Funding for Public 
Schools 


ratio is 8.2:1. For the second year in a 
row, Idaho schools have a better 
student to computer ratio than the 
national average. Other technologies 
are being inipieniented at 
approximately the same rates as the 
nations averages with the possible 
exception of school districts with 
satellite receivers .... Teadiers 
report that their greatest training need 
is in software selection and 
implementation. Second most 
frequently reported need is using the 
Internet. 


Idaho State 


2 


(3) 


Evaluation of the 


The intent of the legislation has been 


Board of 




Idaho State 


Idaho Education 


attained or progress has been made 


Education 




Board of 


Technology 


toward that attainment .... ICTL’s 


12/16/97 




Education 


Initiative for FY 
1997 


ability to measure impacts or benefits 
in a consistent nianner continues to be 
troublesome. 


Idaho State 


2 


(4) 


Public School 


1) Technology is helping teachers 


Department of 




Debra J. 


Technology Grant 


foster student learning and is enabling 


Education 




Dirksen, 


Program FY 96 and 


students to improve the quality of their 


1/98 




Christine 
Bauer, David 

Sarah 

Naylor, 

Archie 

George 


FY97 (Years 1.2 
&3): A Statewide 
1 1 

1^ V L&ici u i \Ji 

Impact of 
Technology 
Integration on Idaho 
Teachers and 
Students 


work; 2) Teachers who use technology 
encourage more collaborative, 
interactive and welT regulated 
learning among students; 3) 
Technology has expanded the range of 
available educational resources; 4) 
Teachers use technology to develop 
students' basic skills and problem- 
solving skills; 5) Teachers need 
training and software to help them 
integrate computers into their 
instructional practices; 6) Districts 
need financial support for technology 
maintenance, repairs, and upgrades. 



Quality Rating Scale : 3 = Exemplary Research/Evaluation Study, 2 = Credible Research/Evaluation Study 
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Table 2.1: Credible Research and Evaluation Studies 



Affiliation 

Date 


Quality 
Rating V 


'^’^port^#^‘ ■ ^ 

Author 


Title 


Results or 
Conclusions 


Idaho State 


3 


(5) 


The Idaho 


1) There is a positive relationship 


Division of 




Idaho State 


Technology 


between academic performance in 


Vocational 




Division of 


Initiative: An 


language math and reading and 


Education, Idaho 




Vocational 


Accountability 


student technology exposure; 2) The 


State Department 




Education, 


Report to the Idaho 


defacto use of the Internet as a state 


of Education, 




Idaho State 


Legislature on the 


network has resulted in technological 


Bureau of 




Department 


Effects of Monies 


compatibility among schools and 


Technology 




of 


Spent Through the 


districts; 3) Colleges of Education and 


Services 




Education, 


Idaho Council for 


School Districts have worked together 


1/1/99 




Bureau of 

Technology 

Services 


Technology in 
Learning (The 
“Accountability 
Report”) 


to prepare teachers to use technology; 
4) Community members have made 
investments in and have been involved 
in the implementation of technology in 
schools; 5) Schools are using 
technology to improve their 
administrative efficiency; 6) Students 
are being trained to install, maintain 
and support technology. 



1) Eighth grade students who reported 
high exposure to technology 
experienced additional gains of 2.7 
months in math, 2.1 months in 
reading, and 1 month in language 
compared to students who had low 
exposure to technology; 2) Eleventh 
grade students who reported high 
exposure to technology experienced 
additional gains of 1 month for math 
and language, and less than a month 
for reading compared to students who 
had low exposure to technology; 3) 
After controlling for socioeco/iomic 
status, smaller additional gains of 1.4 
months in math, 1.1 months in 
reading, and 1.4 months in language 
were reported for the eighth grade 

rrr'nttn anW 1 in r'octriinrr tx/oro 

reported for the eleventh grade group. 



Boise State 


2 


(7) 


Effects of 


All students improved in technology 


University 




Audrey C. 


Technology and 


self-efficacy, objective knowledge, and 


No Date 




Rule & 
Manuel T. 
Barrera 


Curriculum 
Integration on 3”^ 
grade Students’ 
Technology, 
Thinking, and 
Science Process 


vocabulary skills 








Skills 





Boise State 


3 i (6) 


The Relationship 


University, Idaho 


; Clifford 


Between 


State Division of 


i Green 


Technology 


Vocational 




Exposure and 


Education 




Academic 


4/99 




Improvement in 






Idaho’s K-12 Public 






Schools 



Quality Rating Scale : 3 = Exemplary Research/Evaluation Study, 2 = Credible Research/Evaluation Study 
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Table 2.1: Credible Research and Evaluation Studies 



Affiliation 

Date 


Quality 

Rating 


Report # 
Author 


Title 


Results or 
Conclusions 


Boise State 
University 

No Date 


2 


(8) 

Del Siegle, 
Theresa 
Foster & 
Bruce Bedell 


The Impact of 
Presentation 
Software on 
Secondary Science 
Students’ 
Achievement and 
Attitudes 


The study demonstrates that exposure 
to multimedia and presentation 
software increases student 
achievement in a high-level science 
course such as Anatomy and 
Physiology 


Boise State 
University 

1966 


3 


(9) 

Dawn Stram 
Stratham & 
Clark R. 
Torell 


Computers in the 
Classroom: The 
Impact of 
Technology on 
Student Learning 


1) On their own, computers do not 
improve student learning; 2) When 
used appropriately in combination 
with targeted curriculum, instruction, 
and assessnient practices, computer 



technology has been consistently 
shown to make a difference in student 
learning; 3) There are relatively few 
evaluations in the educational 
literature that are based on empirical 
data rather than impressionistic 
reports; 4) Computers appear to be an 
especially effective instructional 
methodology with at-risk students; 5) 
Meta-analyses provide impressive 
evidence for the impact of computer- 
based instruction on student 
achievement. 



Boise State 
University 

4/99 


2 I (10) 

j Deborah E. 
: McGrath 


The Relationship 
between Computer 
Use, Student 
Academic 
Achievement and 
Student Perceived 
Technology 
Competency 


Boise State 


2 1 (11) 


The Status of 


University 


I Del Siegle & 


Vallivue 


No Date 


: Theresa 


Consortium Teacher 




! Foster 


Attitudes Toward 






Technology in the 






Classroom 



1 ) There was a small but statistically 
significant positive correlation 
between home computer use and 
achievement gains in reading, 
language, and mathematics for Idaho 
eighth and eleventh grade students; 2) 
There was a strong positive 
correlation between home computer 
use and perceived technology 
competency 

1) Vallivue Consortium teachers 
believe that including computers in 
their classroom instruction increases 
student learning and provides 
additional educational opportunities 
for students; 2) Vallivue Consortium 
teachers are eager to receive 
additional technology training. 



Quality Rating Scale : 3 = Exemplary Research/Evaluation Study, 2 = Credible Research/Evaluation Study 
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Table 2.1: Credible Research and Evaluation Studies 



Affiliation 

Date 


Quality 

Rating 


Report # 
Author 


Title 


Results or 
Conclusions 


Boise State 
University 

No Date 


2 


(12) 

Manuel T. 
Barrera, III, 
Audren C. 
Rule, C 
Jolene 
Dockstader 
& John A 
Derr 


Comparing 
Technology Skill 
Development in 
Computer Lab 
versus Classroom 
Settings of Two 
sixth Grade Classes 


1) Students learned computer skills 
more effectively in the computer lab 
setting; 2) Students' attitudes toward 
computers were equivalent in both 
instructional settings. 


Challis High 
School, Challis 
Joint School 
District 

7/99 


2 


(13) 

Jennifer 

Jarvis, 

Robert 
Rogers, Julie 
Oerke, Anne 
Lane & Kay 
Piva 


Is Technology More 
Effective in the 
Memorization of 
Multiplication Facts 
Than Traditional 
Methods 


There were no difference in change 
scores between students who used 
software and students who used paper 
flash cards to learn math facts. 


North Gem 
Elementary, 
North Gem 
School District 
7/99 


2 


(14) 

Samuel 

Wadsworth 


Can the Use of Drill 
and Practice 
Software Take the 
Place of Written 
Practice in 
Grammar? 


Students who used the computer to 
practice grammar scored the same on 
a grammar test as students who 
completed textbook exercises with 
paper and pencil. 


Gooding Middle 
School, Gooding 
joint School 
District 

7/99 


2 


(15) 

Cinci Canine 


Does the 
Skillbuildine 
Keyboarding 
Software Help 
Students Increase 
Their Words Per 
Minute and Typing 
Accuracy 


The Skillbuildins software was 
effecting in increasing students' 
keyboarding skills and accuracy. 


Jefferson 
Elementary 
School, Jerome 
School District 
7/99 


2 


(16) 

Afton 

Diemart 


Can First Graders 
Learn Keyboarding 
Skills? 


The majority of students were capable 
of learning proper keyboarding skills. 


MacKay 
Elementary 
School, Mackay 
Joint School 
District 

7/99 


2 


(17) 

Christine 

Hoover 


Will Third Grade 
Students Improve 
Multiplication Skills 
Using Soeedmath 
Software Better than 
They Would Using 
Traditional 
Instructional 
Methods? 


There was an overall improvement in 
students' performance in both time and 
computational accuracy. 



Quality Rating Scale : 3 = Exemplary Research/Evaluation Study, 2 = Credible Research/Evaluation Study 
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Table 2.1 : Credible Research and Evaluation Studies 



, Affiliation^ . 

! y ■ 


Quality/: 

^Rating 


; Report # 
Author 


Title 


^ Results or 1 

Conclusions 


Filer Elementary 
School, Filer 
School District 
7/99 


2 


(18) 

Janice Clark 


Can Identification 
of Rhyming Words 
be Improved by 
Using Computer 
Technology for Drill 
and Practice? 


The average student reading score 
increased between pre-test and post- 
test 


A. W. Johnson 
Elementary 
School, Firth 
School District 


2 


(19) 

Gayla 

Coombs 


Will Mammoth 
Meltdown help 
Students Learn 
Adverbs and 
Adjectives? 


The degree of improvement shown by 
these students has also occurred in 
prior years when the teacher taught 
the same concepts without using the 
software. Consequently, the author 
concluded that using the software did 
not enhance student learning. 


Mackay 
Elementary 
School, Mackay 
Joint School 
District 

7/99 


2 


(20) 

Bev 

Crawford 


Will How the West 
Was One + Two x 
Four help Second 
Grade Students 
Master Basic Order 
of Operations in 
Mathematics? 


The software was an important part of 
the instructional approach which also 
included lecture, demonstration and 
practice with paper and pencil. 


Idaho State 
University 

6/9/99 


3 


(21) 

Albert W. 
Strickland & 
Jack A. 
Coffland 


Mathematics 
Technology Project 


1) Fourth graders mathematics 
achievement scores can be improved 
within an academic year by using an 
instructional model combining 
diagnostic student assessment, 
precision teaching, hands-on teaching 
aids, and computer technology to 
assist the practice and application of 
math skills: 2) The level of 
improvement depends on the 
coni/nitnie/it of the school system, the 
building principal, and the classroom 
teacher. 


University of 
Idaho - Moscow 


2 


(22) 

John Davis 
& Michael 
Odell 


An Evaluation of 
Traditional and 
Internet Approaches 
to Teaching 
Teachers about 
Educational 
Technology 


Teachers taught to use educational 
technology via the Internet learned as 
much as teachers taught using 
traditional means or traditional means 
supplemented by Internet practice. 



Quality Rating Scale : 3 = Exemplary Research/Evaluation Study, 2 = Credible Research/Evaluation Study 
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Table 2.1: Credible Research and Evaluation Studies 



Affiliation 

Date 


Quality 

Rating 


Report # 
Author 


Title 


Results or | 

Conclusions 


University of 
Idaho “ Moscow 


3 


(23) 

Jerome 

Reinger 


A Longitudinal 
Study of Third 
Grade Achievement 
Test Scores Over a 
Three-year Period at 
a Year-round School 
of Choice Using the 
Teaching and 
Learning with 
Computers Method 
of Instruction 


A combination of technology, parental 
choice, and year-round education may 
increase students' achievement on 
vocabulary, language and reading 
tests. 



Quality Rating Scale : 3 = Exemplary Research/Evaluation Study, 2 = Credible Research/Evaluation Study 



Reports (1) through (6) were structured by the goals of the Idaho Educational 
Technology Initiative and provide explicit information about the Initiative. These studies were 
conducted in response to legislative mandate or to respond to Legislator’s questions about the 
activities conducted and progress made as a result of the Technology Initiative. They document 
the infusion of computer hardware into Idaho schools, the training efforts being made by Idaho 
colleges of education, and the ways in which some teachers are changing their instructional 
programs to take advantage of computer technology. One report (5) used Idaho statewide 
assessment data to examine the relationship between student computer use and achievement 
gains, and found that students who had greater technology exposure gained more in tested 
achievement than students with less exposure. This report is of particular interest because of 
the data set generated and the methodology used. The existing data set could be subjected to 
more sophisticated statistical analyses of the relationship among computer use, school and 
district characteristics, academic achievement and socioeconomic status. A similar methodology 
could be used in future years to collect information about classroom technology use throughout 
the state. 

One report (3) stands out from the others in this first group by calling for the definition 
of measurable goals and the creation of statewide performance indicators to track progress 
toward these goals. Although I support this general concept, I caution against the hasty creation 
of simplistic performance indicators or the creation of a system that could impede district 
autonomy and the ability to use technology consistent with district instructional priorities. In 
addition, one must avoid the facile linking of student technology use to academic achievement. 
As the studies reviewed below illustrate, computer technology must be considered within the 
instructional context in which it is used. Technology is one tool that can support learning; it is 
not the only tool, nor always the most effective one. 

Reports (7) through (12) were conducted by teachers and students at Boise State 
University and focus on the acquisition of content knowledge as well as computer skills. Most 
notable is report (9) which presents a major review of the impact of computers on student 
learning. This research demonstrates that technology can add value to classroom instruction 
when used in combination with specific curriculum, instruction, and assessment practices. 
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Report (10) is methodologically interesting because it uses Idaho state assessment data to 
examine the relationship between home computer use and gains in academic achievement as well 
as computer competency. 

Reports (13) through (20) are action research studies conducted by teachers affiliated 
with the Idaho State University “Just in Time” Technology Challenge Grant. Although the 
primary purpose of teacher action research is to improve the instructional skills and judgment 
of the teacher conducting the research, the studies listed above present interesting and credible 
findings on topics of key interest to classroom educators. Results should be generalized 
cautiously (if at all) beyond the students, schools and teachers involved in the research studies. 

It may be useful to consider why these seven studies present credible research results 
while other action research studies (Reports 29 - 39) conducted under the auspices of the “Just 
in Time” Technology Challenge Grant did not. In general, the credible studies address 
researchable questions that are carefully defined. These studies also employ research designs that 
can provide the information necessary to answer the questions posed and which exclude 
extraneous factors that can influence research results. In contrast, the action research studies 
numbered 29 to 39 in Appendix A frequently make claims about the effectiveness of a particular 
program, but do not provide adequate data about what would happen to students if they did not 
participate in the particular program. If action research is to be used to provide credible 
information about technology integration, it appears that teachers will need more guidance about 
how to craft research questions and design a data collection strategies. 

Study (21), The Mathematics Technology Project, is worthy of recognition on several 
respects. First, it emphasizes the role of curriculum and instruction in fostering student learning; 
the use of technology is carefully designed to support this focus. Such an orientation - where 
technology is seen as a part of curriculum and instruction rather than the dominating 
educational force - is necessary if we are to learn how to use technology to extend student 
learning. Second, the study employs a contrast group of teachers who taught students similar to 
those in the treatment group. These contrast teachers continued to teach mathematics as they 
always had done, thus allowing the researcher to draw conclusions about the impact of the 
curriculum, instruction, and technology being used in the treatment group. Finally, the study 
included well-designed achievement measures administered before and after the treatment. 
Although there are some methodological improvements that can be made in future studies, the 
overall approach represented by this research is definitely worth emulating. 

Study (22) provides information about the impact of Internet based teaching on student 
learning , and study (23) portrays the combined impact of technology and school organizational 
factors on student achievement. 
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Conclusions: Learning from Idaho Technology 
Integration Studies 



Looking across the studies, there are a number of conclusions that can be drawn: 

• “Formal” evaluations of the Idaho Educational Technology Initiative provide 
convincing information about the Initiative’s implementation. Schools have 
purchased and installed computer hardware. Teachers are receiving training from 
School Districts and Colleges of Education. Additional teacher training appears to 
be necessary. 

• There is no “accountability mechanism” in place to provide objective data about the 
continuing impact of the Idaho Educational Technology Initiative on teacher 
instruction and student learning. Although statewide performance indicators can 
be used to monitor impact, they must be defined carefully to ensure that they are 
educationally valid and responsive to local conditions and instructional priorities. 

• There is evidence that some teachers who integrate technology in their teaching 
have a positive impact on student learning, and that Integrated Learning Systems 
(e.g., computer managed drill and practice) can also enhance student learning. In 
general, curriculum and instructional practices are as important as (and possibly 
more important than) technology integration in accounting for student learning 
gains. 

• There is evidence that students who have more exposure to computers and who use 
educational technology more consistently gain more in tested academic 
achievement than students who have less exposure to computers and educational 
technology. It is not completely clear how student socioeconomic status and school 
characteristics are related to this finding. 

These research and evaluation results demonstrate that the integration of computer 
technology into classroom curriculum and instructional practices can have a positive impact on 
students' learning and school experience. However, the topics they address - with the exception 
of the formal, implementation studies - appear to be chosen at random. Studies reflect the 
interest and inclinations of the researcher. While this model is appropriate for university-based 
research, it is not well-suited for policy research seeking to identify key factors responsible for 
enhancing student learning. A more efficient approach would be to define in specific and 
measurable ways the goals and processes of technology integration and establish data collection 
mechanisms and strategies that enable the collection of needed information. The dissertation 
research conducted by Clifford Green, and the mathematics research reported by Al Strickland 
and Jack Coffland, provides two examples of how this might occur; there are also other possible 
models. I will return the discussion of efficient, policy-relevant research in the final chapter of 
this report. 
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Endnotes 



1. Idaho State Department of Education, Connections: A Statewide Plan for Technology in 
Idaho Public Schools. Boise ID: Author, p. 6. 

2. It can be argued that I have been overly rigorous - that it is unrealistic to judge an action 
research project conducted by a first-grade teacher using the same criteria applied to a 
university-based, PhD-enobled researcher. I think this argument has merit, but it is 
irrelevant to the charge I was given. The purpose of this analysis, as outlined in the 
contracted scope of work, is to: 

Take a fresh look at the studies referenced in the Accountability Report and 
consider their methodological rigor, their generalizability to students 
throughout Idaho, and the force of their conclusions. 
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Chapter 3: ICTL Goals and the 
Experience of Idaho Educators: A Report 
on Site Visits to 1 6 Idaho School Districts 

Plan for the Chapter 

This chapter reports my observations of how Idaho schools are achieving the goals of the 
Educational Technology Initiative. It is based on visits to 16 Idaho school districts and three 
colleges of education. In describing the results of my visits, I will first explain how I chose the 
districts to be visited, and the strategies I used to gain as much information as possible during 
the visits. I will then organize my observations according to ICTL goals, and conclude with some 
summary thoughts about the implementation progress of the Idaho Educational Technology 
Initiative. 

School District Selection 

I asked the Bureau of Technology Services (BOTS) , Idaho State Department of Education 
for a preliminary listing of school districts. I requested that the districts vary according to region, 
size, and the degree (just beginning, some strides made, a leader in educational technology use) 
to which they had integrated technology into their instructional program. BOTS provided me 
with a list of 30 districts in all 5 regions of the state. 

Based on that preliminary listing, I selected 16 school districts that varied according to 
the previous criteria and which could be visited within the budget and time constraints of this 
evaluation. These school districts represent a purposive sample, and give a good indication^ I 
believe, of the types of things that are being accomplished with educational technology 
throughout Idaho. Table 3. 1 displays the characteristics of the school districts selected for visits. 
It is worth noting that these districts make up fourteen percent of Idaho’s 112 school districts. 



Table 3.1: Characteristics of School Districts Visited 



Degree of 
Technology 
Integration 


# of Districts 


District Size 


# of Districts 


Region 


# of Districts 


Leader 


6 


0-500 


3 


1 


2 


Some Strides 


4 


501-1,000 


3 


2 


4 


Beginning 


6 


1,001-5,000 


5 


3 


4 






5,001-10,000 


3 


4 


2 






10,001 and up 


2 


5 


1 










6 


3 
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Site Visit Strategy 



Table 3.2 displays the schools I visited along with the district name, number, county, and 
November 1997 enrollment. 

To arrange district visits, I first contacted the Superintendent by phone and explained the 
purpose of my tour. I asked for the opportunity to see at least one elementary and one secondary 
school, and to talk briefly and informally with some teachers. I also asked that any relevant 
district documents such as Technology Plans be sent to me in advance of my arrival. 

Superintendents welcomed the chance to show me how their districts were using 
educational technology. They went out of their way to fit into my visit schedule and to arrange 
for the District Technology Coordinator and other staff to be available to talk with me. Each visit 
lasted approximately 3.5 hours. This gave me time to tour facilities, visit classes, and speak with 
Superintendents, Technology Coordinators, teachers, and students. I took notes as we walked 
around and then expanded those notes into a narrative account at the end of the day. 

Colleges of Education 

I met with each of the Regional Technology Advisors and learned of their activities and 
impressions of Idaho schools. I also spoke with deans and other staff members involved in 
technology training and research at the University of Idaho-Moscow, University of Idaho-Coeur 
d’Alene, Idaho State University, and Boise State University. These meetings lasted approximately 
2.5 hours, and I was provided with a variety of printed information describing preservice and 
inservice technology training programs. I took notes during these meetings and expanded them 
later into narrative summaries. 

Accomplishing ICTL Goals 

Following the passage of HB901, the Idaho Council for Technology in Learning defined 
eight goals to guide the Idaho Educational Technology Initiative. There were: 

• Goal 1: Integration - to improve the quality and effectiveness of classroom 
instruction and learning for all learners by integrating technology-based resources in 
conjunction with state curriculum guides; 

• Goal 2: Compatibility - to ensure the compatibility of technology-related equipment 
to facilitate a comprehensive statewide network system; 

• Goal 3: Collaboration with Colleges of Education - to facilitate collaboration with 
the colleges of education in the preparation and inservice training of teachers for 
integration of technology into instructional practices; 

• Goal 4: Community Collaboration - to encourage the collaboration of schools, 
libraries, community members, state agencies, organizations, businesses, industries, 
and post-secondary institutions to meet the needs of all learners; 
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Table 3.2: Schools Visited 



School District 


# 


County 


District 


Schools Visited 








Enrollment 





South Lemhi 


: 292 


Lemhi j 


171 


Leadore School 


Mullan 


: 392 


Shoshone i 


175 


John Mullan Elementary School 
Mullan Junior-Senior High School 


Kamiah Joint 


i 304 


Lewis : 


674 


Kamiah Elementary School 
Kamiah Middle School 
Kamiah Senior High School 


Whitepine Joint 


j 286 


Latah i 


683 


Troy Elementary School 
Troy Junior-Senior High School 
Deary School 


Gooding Joint 


: 231 


Gooding : 


1,333 


Gooding Elementary School 
Gooding Middle School 
Gooding High School 


Salmon 


j 291 


Lemhi j 


1,359 


Pioneer Elementary School 
Salmon Junior High School 
Salmon High School 


Orofino Joint 


1 171 


Clearwater i 


1,660 


Orofino Elementary School 
Orofino High School 


Payette Joint 


j 371 


Payette i 


1,949 


Westside Elementary School 
Payette High School 


Kuna Joint 


! 3 


Ada : 


2,608 


Ross Elementary School 
Kuna High School 
Kuna Evening School 


Jerome Joint 


j 261 


Jerome • 


3,144 


Horizon Elementary School 
Jerome High School 


Blackfoot 


: 55 


Bingham : 


4,449 


Ridgecrest Elementary School 
Groveland Elementary School 
Mountain View Middle School 


Lewiston Independent 


j 340 


Nez Perce 


5,172 


McGhee Elementary School 
Jennifer Junior High School 
Lewiston Senior High School 
Tammany Alternative Learning 
Genter 


Coeur d’Alene 


i 271 


Kootenai • 


8,742 


Fernan Elementary School 
Lakes Middle School 


Nampa 


j 131 


Canyon i 


9,573 


District Office 


Pocatello 


j 25 


Bannock j 


13,127 


Wilcox Elementary School 


Meridian Joint 


; 2 


Ada ; 


20,778 


Andrus Elementary School 



Lowell Scott Middle School 
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• Goal 5: Technology Systems - to create secure technology systems that enhance 
the efficient operation of schools; 

• Goal 6: Evaluation ~ to plan, evaluate and publicize the impact of technology on 
teaching, learning, resource utilization, and the efficient operation of schools; 

• Goal 7 : Student Training - to train students in the installation, maintenance, and 
support of technology systems; and 

• Goal 8: Systems Support - to provide district-wide support structures for 
training and for the installation, maintenance and support of technology systems. 

In the remainder of this chapter I will describe my observations regarding the experience 
of Idaho schools and school districts in accomplishing these goals. 

Goal 1: Integration. Integrating educational technology into the daily workings of 
classrooms and the instructional strategies of teachers is an issue that is on the mind of Idaho 
administrators and teachers. When I would ask Technology Coordinators, principals or 
superintendents about the three biggest challenges they faced, one challenge was inevitably 
“technology integration.” 

Idaho educators are not only talking about technology integration, they are also doing 
it. I observed numerous examples of teachers at all grade levels whose students use the Internet 
to prepare traditional reports and Power Point slide shows. I saw elementary and middle school 
teachers shepherding students through Accelerated Reader and Accelerated Math programs. I 
enjoyed science lectures peppered with Power Point slides, and clicked through Hyper studio 
stacks about Idaho history. Unquestionably, there is a core of Idaho teachers who are committed 
to and proficient in the integration of technology within their classroom instructional programs. 
These innovators provide models for (what I expect is) a larger group of teachers who are 
interested in incorporating technology into their teaching, but have not had time, or skill or 
confidence to develop technology-infused lesson plans. This will happen. My sense is the 
momentum to integrate technology into daily instruction is present and building. New 
expectations are being set. Teacher evaluation systems and district scope and sequence 
documents are being altered to incorporate technology. 

Although my observations provide evidence that technology is being integrated into 
instruction on a daily basis, they provide no basis for estimating how many of Idaho’s 15, 000 
teachers are inveterate technology integrators, and what training and support would enable 
others to join their ranks. Nor do my observations allow judgments of how frequently teachers 
employ technology in the course of their teaching. Further, quantitative information on these 
questions will be available in February when the results of a statewide survey of Technology Use 
and Training Needs are analyzed. 

Goal 2: Compatibility. The absence of a state backbone has led Idaho school districts to 
turn to the Internet for communication. This, in turn, has ensured compatibility among districts 
as information is transmitted using Internet protocols. Consequently, the ICTL goal of 
compatible systems and information has been achieved. 
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Goal 3: Collaboration with Colleges of Education. The educators with whom I spoke 
were consistently complimentary about the service and support they had received from the 
Regional Technology Advisors and other staff at the University of Idaho (both Moscow and 
Coeur d’Alene), Lewis-Clark State College, Idaho State University, and Boise State University. 
Some also spoke highly of training received from Northwest Nazarene College and Northern 
Idaho College. Table 3.3 displays the collaborations mentioned by different school districts 
during my visits. 

Technology trainers were praised as being, at various times, supportive and demanding. 
Teachers spoke appreciatively of the individual attention they had received and trainers’ 
willingness to work with them and their students in their own classrooms. While two 
Technology Coordinators spoke of conflicts between themselves and the Regional Technology 
Advisor, these disputes were eventually resolved. 



Although there are a variety of strategies being used by the colleges of education to 
support the Idaho Educational Technology Initiative, it appears that all of the strategies are 
paying off, and all have their vocal and fervent supporters. I am struck by the enormity of the 
challenge they face; it is, in the words of Heidi Rogers, an “insurmountable opportunity.” There 
are approximately 15,000 teachers in the State of Idaho, and a training staff at the colleges of 
education of approximately 20. It is clear that these few individuals cannot directly meet the 
technology needs of diverse teachers in geographically dispersed districts. The challenge is to 
mount and continue technology training programs that produce teachers who can train and 
support other teachers within the same building and district. The BSU Train the Trainers 
program and the LCSC Mentor Teacher Project are two examples of such a training strategy. 



Goal 4: Community Collaboration. Table 3.3 describes the many different ways Idaho 
school districts are using technology to collaborate with the larger community. Community 
members consistently sit on district and school technology committees. Businesses frequently 
donate cast-off computers to schools. School libraries are being opened to the public after school, 
so community members can access the Internet. Technology fairs and parent nights are being 
scheduled to introduce the community to the technology students use each day. 



Technology also provides the means for some collaborations as in the VALNET Library 
Network which hosts a common catalog, indexing books found in both school and community 
libraries. Another example of technology-based collaboration is provided by Camusnet, a locally- 
owned, non profit Internet service provider that shares equipment, and connectivity with the 
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institutions and local businesses. 



Several districts shared plans for future community and parent collaboration using the 
World Wide Web. They are in process of developing web sites that contain school assignments, 
sports schedules, teacher email addresses and password protected student information including 
grades, attendance, and test scores. 
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Table 3.3: School District Activities for Goals 3-5 



School 

District 


Goal 3; 
Collaboration 
with Colleges 
of Education 


Goal 4: 
Community 
Collaboration 


Goal 5: 
Technology 
Systems 


^ South Lemhi 


University of Idaho 
College of Southern Idaho 
ISU Challenge Grant 


Community Attends 
Computer Training 
Classes 

Community Members on 
Tech Committee 
Loan of Old Computers 
to Community Planned 
Salmon Public Library 


Satellite Internet Access 
School Pro Student 
Database 
Meal Planning 
Computerized Library 
Catalog 


Mullan 


University of Idaho- 
Moscow 

Northwest Nazarene 
College 


School-based Enterprise 
Business Partnership 
Community Television 
Station 

Community Members on 
Tech Committee 
VALNET Library Network 


District Network 
Email for teachers and 
Administrators 
EPES Student Database 
Computerized Library 
Catalog 


Kamiah Joint 


LCSC Mentor Program & 
Teacher Training 
LCSC Extended Learning 
Program 


Computer Donations 
Community Members on 
Tech Committee 
VALNET Library Network 
Community Internet 
Access with Camasnet 
Member of Clearwater 
Valley Advisory 
Committee 


District Network 
SASI Student Database 
Email for teachers and 
Administrators 
Computerized Library 
Catalog 


Whitepine 

Joint 


ISU Challenge Grant 
University of Idaho-Coeur 
d’Alene 


Community Members on 
Tech Committee 
Community Television 
Station 

Computer Donations 
VALNET Library Network 


District Network 
Email for teachers and 
Administrators 
Video Conferencing 
Capability 

School Master Student 
Database 




College of Sotithcrn Idaho 


CoiTiputer Donatioiis 


District Network 
SASI Student Database 
Computerized Library 
Catalog 


Salmon 


ISU Challenge Grant and 
Teacher Computer 
Training 


Computer Donations 
Community Members on 
Tech Committee 
Salmon Public Library 


District Network 
SASI Student Database 
Mac School Student 
Database 

Computerized Library 
Catalog 
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Table 3.3: School District Activities for Goals 3-5 



School 

District 


Goal 3: 
Collaboration 
with Colleges 
of Education 


Goal 4: 
Community 
Collaboration 


Goal 5; 
Technology 
Systems 


Orofino Joint 


University of Idaho-Coeur 
d’Alene 

LCSC Mentor Program& 
Shared Computer Lab 
Community Internet 
Access with LCSC 


Community Members on 
Tech Committee 
VALNET Library Network 


Email for teachers 
and Administrators 
in Wired Buildings 
District Designed Student 
Database 

Computerized Library 
Catalog 


Payette joint 


BSU Distance Learning 
Project 

BSU Train the Trainers 
University of Idaho 


Community Members on 
Tech Committee 
Computer Donations 


District Network 
School Master Student 
Database 

Email for teachers and 
Administrators 


Kuna joint 


BSU Train the Trainers 


Kuna Community Library 
Community Members on 
Tech Committee 
School-based Enterprise 
(Prestige Publications) 
Senior Computer Classes 
Planned 

Community Attends 
Computer Training 
Classes 

Computer Donations 


District Network 
Email for teachers and 
Administrators 
School Master Student 
Database 

Computerized Library 
Catalog 


Jerome joint 


ISU Challenge Grant 


Technology Newsletter 
Distributed to 
Community 
Computer Donations 
Community Members on 
Tech Committee 
Community Use of 
Computer Labs 
Partner with Jerome 
Recreational District 


District Network 
Email for teachers and 
Administrators 
Video Conferencing 
Capability 
Discipline Specific 
Technology 
Classrooms 
SASI Student Database 
Computerized Library 
Catalog 


Blackfoot 


ISU Challenge Grant and 
Teacher Training 


Community Technology 
Center Planned 
Community Members on 
Tech Committee 
Technology Projects 
Involve Community 
Members 


District Network 
Email for teachers and 
Administrators 
SASI Student Database 
Computerized Library 
Catalog 
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Table 3.3: School District Activities for Goals 3-5 



School 

District 


Goal 3: 
Collaboration 
with Colleges 
of Education 


Goal 4: 
Community 
Collaboration 


Goal 5; 
Technology 
Systems 


Lewiston 


LCSC Mentor Program & 


Community Members on 


District Network & 


Independent 


Teacher Training 


Tech Committee & 


Updated Phone System 




University of Idaho 


Professional Technical 


Email for Teachers, 




Teacher Training 


Academy Advisory 
Committees 

VALNET Library Network 
High School Library 
Opened to Community 
at Night 


Administrators & 
Classified Staff 
SASI Student Database 
Mac School Student 
Database 

Voicemail for High School 
teachers & 
Administrators 
Computerized Library 
Catalog & 
Transportation and 
Food Service Planning 


Coeur d’Alene 


LCSC Mentor Program 


Community Attends 


District Network 




University of Idaho-Coeur 


Computer Training 


Email for teachers and 




d’Alene 


Classes 


Administrators 




Northen Idaho College 


Partnerships with Idaho 


Skyward Student Database 




ISU Challenge Grant 


Fish and Game, Bureau 
of Land Management 
and Forest Service 
School Technology Night 
Community Members on 
Tech Committee 
VALNET Library Network 
Partners with Harpers 
Manufacturing, US 
Bank, Kootnet 
Consortium, Riverbend 
Professional Technical 
Academy 

Chamber of Commerce 


Web Speed 
Grade Machine 
Computerized Library 
Catalog 






Technology Fair 




Nampa 


BSU Train the Trainers 


Partners with Zilog & 
Micron 

Computer Donations 
Community Members on 
Tech Committee 


District Network 
SASI Student Database 
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Table 3.3: School District Activities for Goals 3-5 



School 

District 


Goal 3: 
Collaboration 
with Colleges 
of Education 


Goal 4; 
Community 
Collaboration 


Goal 5: 
Technology 
Systems 


Pocatello 


ISU Challenge Grant and 
Teacher Training 


Computer Donations 
Community Members on 
Tech Committee 
Community Education 
Program 

Business Partnerships 


District Network 
SASI Student Database 
Email for teachers and 
Administrators 
Automatic Student Absence 
Phone Calls & Letters 
Computer Automated 
System for Substitute 
Teachers 

Computerized Library 
Catalog 


Meridian 


BSU Train the Trainers and 


Computer Donations 


District Network 


Joint 


Teacher Computer 
Training 

LCSC Teacher Computer 
Training 


High School Libraries 
Opened to Community 
After School 


District Database with 
Student Attendance, 
Demographic, 
Transportation & 

Food Service 
Information 

Automatic Student Absence 
Phone Calls & Letters 
Email for teachers and 
Administrators 



Goal 5: Technology Systems. Table 3.3 exhibits a number of different ways school 
districts have used technology to make their operations more efficient. Most commonly, schools 
have installed a wide area district network, and issued email accounts to all administrators and 
teachers. Many districts also provide email for classified staff. All districts also benefit from the 
use of commercially produced or district designed student data bases. These make it easier to 
maintain attendance, discipline, health, student status, and grade records. The more advanced 
of these systems allow teachers to input student attendance and grade information on their 
classrooni computers i the software then compiles this information and produces attendance 
reports and report cards. Some systems automatically produce letters informing parents that 
their children have been absent or tardy or place calls to the student’s home. 



Goal 6: Evaluation. Table 3.4 displays the technology evaluation activities recounted by 
school district staff during my district visits. Most typically these activities consist of teacher 
surveys focusing on technology competencies and training needs. Occasionally districts 
employed internal or external evaluators to examine the outcome of a specific technology 
project, or analyzed their curriculum to see where technology integration might make a 
difference in student learning, but overall, such evaluation activities were infrequent. Looking 
across all the ICTL goals, the goal of evaluation appears to be the one for which there has been 
the least progress toward attainment. 
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Table 3.4: School District Activities for Goals 6-8 



School 

District 


Goal 6; 
Evaluation 


Goal 7: 
Student 
Training 


Goal 8: 
Systems 
Support 


South Lemhi 


District Survey of Teacher 
Computer 
Competencies 
District Survey of High 
School Graduates 




Teacher Technology 
Mentors 


Mullan 


District Survey of Teacher 
Computer 
Competencies 


TNT Program 
Student Network 
Technician 


District Technical & 
Network Support 
Student Technicians 


Kamiah Joint 


District Survey of Teacher 
Computer 
Competencies 


TNT Program 
MCSE & CCNA 

Certification Program 


District Technical & 
Network Support 
Student Technicians 


Whitepine 

joint 


District Survey of Teacher 
Computer 
Competencies 


TST Program 


District Technical & 
Network Support 
Teacher Technology 
Mentors 


Gooding joint 


District Survey of Teacher 
Computer 
Competencies 


TST program 


District Technical & 
Network Support 
Student Technicians 


Salmon 


District Evaluation of Goals 
2000 Project 

District Survey of Teacher 
Computer 
Competencies 


Planned for Future 


District Technical & 
Network Support 
Teacher Technology 
Mentors 


Orofino joint 


District Evaluation of 
Student Reading and 
Math Achievement 
District Survey of Teacher 
Computer 
Competencies and 
Needs 

District Survey of Student 
Computer Use 


TNT Program 


District Technical & 
Network Support 
Student Technicians 


Payette joint 


District Designed 
Computer-based 
Assessment Program 
Planned 


TST Program 
Student Network & 
Computer 
Technicians 


Building Technology 
Coordinators 
District Technical & 
Network Support 
BSU Trainers (Payette 
Teachers) 

Student Technicians 
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Table 3.4: School District Activities for Goals 6-8 



School 

District 


Goal 6; 
Evaluation 


Goal 7; 
Student 
Training 


Goal 8; 
Systems 
Support 


Kuna Joint 


One Curriculum Area Per 
Year Examined 
Computer Usage Surveys 


TST Program 
A+ , Visual Basic 8i C++ 
Planned 

Student Network 8c 
Computer Technicians 


Classroom Technology 
Coordinators (Teachers) 
District Technical & 
Network Support 
BSU Trainers (Kuna 
Teachers) 

Student Technicians 


Jerome Joint 


District Evaluations of 
Goals 2000 Projects 
District Survey of Teacher 
Computer 
Competencies 
Seven Technology 
Evaluations Now 
Being Conducted 


ARTEC Program includes 
MCP. A+, TST 8c 
Cisco Training 


MCSE Student Technicians 
District Technical & 
Network Support 
Just in Time Challenge 
Grant 


Blackfoot 


District Survey of Teacher 
Computer 
Competencies 
District Curriculum Audit 
District Evaluation of Goals 
2000 Projects 
Evaluation of Mathematics 
Technology Project 


TST program 


District Technical & 
Network Support 
Building Level (Teacher) 
Technician 


Lewiston 

Independent 


Web-based District Survey 
of Teacher Computer 
Use 

District Generated Student 
& Class Idaho 
Reading Inventory 
Reports 

Smart Lab Evaluation 
Personal Staff Individual 
Technology Plans 
Info-tracker 


TNT Program 
MCSE 8c CCNA 

Certification Program 
Technology Graduation 
Requirements 
Smart Lab Requirements 
for Some Schools 


District Technical & 
Network Support 
Student Technicians 
Teacher on Special 
Assignment for 
Technology 
Building Educational 
Technologist 
VALNET Cataloguer 


Coeur d’Alene 


District Survey of Teacher 
Computer 
Competencies and 
Needs 

District Curriculum Audit 
District Technology Audit 
Management and 
Operations Review 
Benchmarks of Technology 
Integration 

External Project and Grant 
Evaluations 


TST Program 
CCNA 8c CNE 

Certification Program 


District Technical 8c 
Network Support 
Technology Facilitators 
Prototypical Technology 
Teachers 

Student Technicians 
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Table 3.4: School District Activities for Goals 6-8 



School 

District 


Goal 6: 
Evaluation 


Goal 7: 
Student 
Training 


Goal 8: 
Systems 
Support 


Nampa 


District Survey of Home 
Computer Technology 
District Surveys of Student 
& Teacher Computer 
Competencies 


TST & A+ Program 


District Technology 
Evangelists (Teachers) 
District Technical & 
Network Support 


Pocatello 


District Survey of Teacher 
Computer 
Competencies and 
Needs 


TST program 

Summer Training Program 
for Student Technology 
Mentors 

Student Network 
Technician 


District Technical & 
Network Support 
Teacher Technology 
Mentors 

Student Technology 
Mentors 


Meridian 

Joint 


District Survey of Teacher 
Computer 
Competencies 


CNE & MCSE Certification 
Program 

Student Interns in Network 
Management 
Web Design Program 


Building Technology 
Trainers 

District Technical & 
Network Support 
Technology Education 
Specialists (Teachers) 
BSU Trainers (Meridian 
Teachers) 

Student Technicians 



There are at least two reasons why evaluation efforts have languished in most districts. 
.First, evaluation is different than teaching or managing. Evaluation requires specialized skills. 
These skills are generally not available in small school districts, and sometimes not available in 
larger districts. Until recently, evaluation models and technical support have not been easily 
available to districts. In the absence of trained evaluators and evaluation training, districts have 
found it difficult to evaluate technology use. 



Second, evaluation takes time and costs money. When Superintendents or Technology 
Coordinators are given the choice between hiring trainers to support teachers as they work to 
integrate technology into their instruction and spending the same funds to hire an evaluator. 



ftr-Sll tTH 

lllV.yOL Will V^l lV./V./<30 0|-/V^11\-1 1.11^ Axi 

priorities will take precedence. 
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In the final analysis. I’m not convinced that school districts could have provided - on 
their own - the kind of evaluation information envisioned in ICTL Goal 6. Here, evaluation is 
defined as: “to plan, evaluate and publicize the impact of technology on teaching, learning, 
resource utilization, and the efficient operation of schools.’’ To my ears, these words suggest data 
that is convincing to an external audience, data that demonstrates that technology expenditures 
have had a positive impact on Idaho education. While such an evaluation can certainly be 
conducted, it lends itself best to a coordinated effort external to individual districts. It requires 
consistent data, collected in a standard fashion, from a representative sample of teachers, schools, 
students, and districts. Such evaluations require careful specification of outcomes, deliberate 
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instrument development, scheduled data collection activities, and extended analysis. They need 
to be orchestrated - and probably conducted - by an external agent. 

School district evaluations are generally most productive when they are undertaken to 
provide information that enables schools to change and improve their own functioning. From 
a districts point of view, technology is simply one strategy among the many used for 
instructional improvement. To isolate this strategy for evaluation and focus more effort on it 
than one would on, say, curriculum development does not seem a good use of whatever scarce 
evaluation resources districts possess. 

Note that I am not decrying analyses of the impact of educational technology on teachers, 
students, and schools. To the contrary, I think it is imperative to examine how technology is 
being used, and to what effect. (I will return to this idea in the final chapter of this report.) My 
point here is simply to argue that such evaluations are best conducted by impartial, external 
agents, instead of local school districts. Teachers and administrators simply do not have the 
necessary time, money, expertise or disinterest to be conduct the type of evaluation suggested 
by ICTL Goal 6. 

Goal 7: Student Training. Table 3.4 displays the opportunities for student technology 
training provided by the Idaho districts I visited. In nearly all districts, students can take classes 
that prepare them to become competent computer technicians. I observed students learning to 
build and repair computers, design and trouble shoot computer networks, and coach teachers 
in computer operation and software. In approximately one-third of the districts, students were 
studying for advanced certification as a Microsoft Certified Systems Engineer, a Certified 
Netware Engineer, or a Cisco Certified Network Associate. 

School-based training programs such as these provide opportunities for students to not 
only gain marketable skills, but also to use these skills to assist district teachers and Technology 
Coordinators maintain and upgrade districts technology. 

Goal 8: Systems Support. As indicated in Table 3.4, all districts had established a formal 
system to provide ongoing technology support to teachers and administrators. In general, this 
consisted of a combination of “central office” support supervised and sometimes coordinated by 
the district Technology Coordinator and “building level” support provided by Teacher 
Technology Mentors and Student Technicians. Although districts had carefully planned and 
revised their technology support system, they consistently reported that they were unable to 
meet current support needs in a timely manner. Districts were unanimous in noting that as the 
technology use of teachers and students increased, they needed more staff, more money and 
more time to support this increased use. 

Several superintendents recounted the electrical and air conditioning upgrades that had 
to be completed before they could install new computer equipment. Many spoke ambivalently 
of the technology they had installed over the past five years. One superintendent told me: 

While we re pleased to have technology in the district, it is an overhang and now 

we have to maintain and upgrade the technology over time; this is a district 

expenditure we haven’t had before. 
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Nearly all Superintendents and Technology Coordinators expressed the concern that Idaho 
Educational Technology Funding would end, and their district would be left with the cost of 
maintaining and upgrading the technology they had already installed. Many administrators 
mentioned the need to have the state funding formula changed to include a Technology 
Coordinator as part of the district allocation. One superintendent put it this way: 

Kids need busses to get to school, and the state gives us money for bus mechanics; 

Kids need to eat, and we get money for cooks. Well, kids need to become 
competent using technology and use technology for learning. We need money for 
a Technology Coordinator. That job is at least as important to a kid’s education 
as a cook. 

Several Technology Coordinators expressed frustration that they did not have the 
resources needed to provide adequate support. One directed me to support cost calculations 
presented on the Web Site of the Consortium for School Networking (http://www.cosn.org/tco/) . 
Here, the Consortium cites a study by the well-known technology consulting firm, Forrester 
Research, nothing that in large corporations, there is one support person for every 50 PCs, at a 
cost of $ 1 4 2 per PC per year. Should school districts adopt this model, one thousand PCs would 
need a staff of 20 and a budget of $1.4 million for support. While this is clearly an unrealistic 
model for school districts to emulate, it does demonstrate that supporting computer and network 
technology is time- and personnel-intensive, and can not be ignored when projecting technology 
expenditures. 

Technology Coordinators spoke of the difficulties they had faced in trying to find - and 
afford - competent staff. Network Administrators were an area of special concern. Two 
Technology Coordinators mentioned advertising a Network Administrator position, and not 
receiving any applicants because of the comparatively low salary being offered. 

Although facilities, hardware, and personnel costs were frequently mentioned as a 
challenge, it was the ability to provide adequate ongoing support to teachers that appeared to be 
the most critical systems support need. One Technology Coordinator told me: 

Look, if Mrs. Jones starts using technology with her class each day, and then one 
day the technology doesn’t work, what’s she going to do? She’s integrated it, she’s 
built her lessons around it, and now it doesn’t work. 

Others spoke of their desire to minimize the obstacles teachers face in using technology. This 
was especially important for teachers that are just now starting to integrate technology into their 
instruction. 

Think of the teacher who has spent time developing her first technology 
integrated lesson plan. She takes the big plunge with her class - and she can’t 
connect to the Internet, or get Power Point to do what she needs it to do. She 
needs help right then, to get her over that hump, and maintain her interest in 
using technology. It’s very easy to go back to doing things the old way. 

In short, supporting the educational technology that had been installed with Idaho 
Educational Technology Funds is worrisome to Superintendents and Technology Coordinators. 
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Uncertain they are providing adequate support today, they question where the people and funds 
will come from to support increased technology use in the future. When asked to look ahead to 
future challenges, they answered with one voice: Supporting and Upgrading the technology we 
already have. 

ICTL Goals and Implementation: 

An Outsider's Perspective 

Evidence of Technology Integration. Looking across the 16 districts I visited this fall, 
it is evident that schools, teachers, and districts are making progress toward accomplishing the 
goals established by the Idaho Council for Technology in Learning. Teachers are integrating 
technology, colleges of education are training teachers, students are learning to be Netware 
Engineers. My general impression is that educators have turned their attention from completing 
the technological infrastructure toward the integration of technology into instruction. 

There are a number of sources of evidence for this conclusion. When I asked 
administrators what they saw as the three greatest challenges to be faced over the next three 
years, technology integration would invariably be mentioned. When I asked Regional 
Technology Advisors and Bureau of Technology Services staff to compare the frequency with 
which they received “integration” versus “technology/connectivity” questions from Technology 
Coordinators and teachers, they indicated “integration” questions about specific software 
packages or lesson plans were more frequently asked than pure “technology/connectivity” 
questions. When I visited schools, I learned of a number of strategies in place to expand 
technology integration from the “pioneering” teachers to their interested, but hesitant colleagues. 
I believe Idaho school culture is changing from a focus on getting and installing the technology 
to using it productively. This should be thought of as a natural evolution. As one administrator 
told me, “If you don’t have it, you can’t use it.” 

Other research studies have documented the same evolution from an emphasis on 
technology to an emphasis on learning. Shelley Goldman, Karen Cole and Christina Syer at the 
Institute for Research on Learning describe their own experiences working with a variety of 
teachers and schools. 



In case after case we see that when computer technologies are adopted, the 
learning about the technology often takes over, and it is only after several rounds 



of integrating technology with content that content emerges in strong wa3/s . . . 
The good news is that content learning does emerge and is very rich once the 
technology recedes as the focus of activities in the classroom.^ 



For this evolution to continue and result in rich content learning, it will be necessary for 
teachers and college of education staff to continue to develop content^specific, instructional 
models that teachers can emulate and adapt to their own circumstances. Classroom 
implementation of new instructional models is demanding and difficult, and teachers generally 
make the best progress when supported by professional development. As one teacher remarked 
when reflecting upon her own attempts at integrating technology into her classes: 
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It s like I’m a new teacher again. I’m trying things I don’t really know how to do. 

It takes time and I make mistakes, but I get better and better at it. My students 

help me and we work at integrating technology together. 

Lagging Goal Attainment: Systems Support and Evaluation. Having made the general 
observation that most ICTL goals are being achieved, and the emphasis in school districts is 
shifting from getting technology for instruction to using technology in instruction, I want to 
comment on two goals for which less progress appears to have been made: Goal 8-Systems 
Support and Goal 8-Evaluation. 

It is not hard to imagine why accomplishment of these goals falls behind that of other 
goals. Conducting educational evaluations and supporting a complex technology system require 
expertise often missing from a school district. Accomplishing these goals at a high level of 
performance also costs school districts more than some of the other ICTL goals (e.g., Goal 2- 
Compatibility or Goal 3-Community Collaboration). 

If my observations are shared by the Regional Technology Advisors and ICTL members, 
it may be worthwhile to consider new or additional strategies that would help Idaho educators 
attain these goals. Systems support might be enhanced by organizing regional training and 
technical assistance centers. It is also possible that network administration could be shared by 
regional administrators supported by school district consortia. This might allow District 
Technology Coordinators to focus less on network administration and more on providing 
classroom teachers and building administrators with hardware and software support. 

I have already suggested that a statewide evaluation should be conceived and conducted 
by an impartial, external evaluator. This could not only reduce the burden on school districts, 
but could result in the systematic collection of more reliable data. It is evident that progress has 
been made accomplishing the goals of the Idaho Educational Technology Initiative. More exact 
assessment will require explicit benchmarks against which progress can be measured. The next 
(and final) chapter of this report presents some guidelines for future evaluations of teachers and 
students educational technology use. 
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NOTES 



1. Goldman, S., Cole, K., & Syer, C. (1999). The Technology/Content Dilemma. Paper 
presented at The Secretary's National Conference on Educational T echnology : Evaluating 
the Effectiveness of Technology. July 12-13, 1999. Washington, DC. 
(http://www.ed.gov/Technology/TechConf/1999/whitepapers/paper4.html) 
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Chapter 4: Assessing the Impact 
of Educational Technology: 
Recommendations for Future Evaluations 



This chapter contains my reflections and suggestions on the enterprise of technology 
evaluation. I hope to convey some perspectives and ideas that will be useful to Idaho 
policymakers as they consider the future of the Idaho Educational Technology Initiative. 

Clarify Evaluation Goals in Advance 
and Determine Data Collection Strategies 

Evaluation is a broad term that includes a variety of activities. While the overall task of 
the evaluator is to make judgments of value and worth, different audiences will be interested in 
different types of Judgments. It is useful to clarify at the outset of an evaluation the sorts of 
Judgments that need to be made, and to define the data and criteria that will be used to make 
these judgments. Unless such planning is done before program implementation is too far 
advanced, an evaluator finds himself with the task of “making the most” of what is available. 

My work in preparing this report has combined “meta-evaluation” (the evaluation of 
others’ evaluations) with site visits and interviews to collect new data about technology 
implementation in Idaho Schools. While this approach has provided useful information, a more 
powerful and efficient approach would have been to set specific evaluation goals at the inception 
of the program and specify the necessary data collection activities at that time. While available 
information can provide helpful insights, it is generally not as complete or incisive as specific 
information, collected over the course of project implementation, and matched to explicit 
evaluation goals. 



Define Program Goals as Explicitly as Possible 



Although not everything of value can be measured, from an evaluator’s - and often from 
a program administrator’s - point of view, it is worthwhile to state program goals as explicitly 
as possible. Such goals can be refined into explicit benchmarks that provide guidance to pi tj^i aiu 



implementors and evaluators alike. While I believe the information contained in this report will 
be useful to Idaho policymakers and educators, I would have been able to provide a more incisive 
assessment of accomplishments and deficiencies had the goals of the Idaho Technology Initiative 
been more clearly specified, and a timetable for their accomplishment defined. 



Consider Technology Use in Context 

Dr. Eva L. Baker is Co-Director of the National Center for Research on Evaluation, 
Standards, and Student Testing (CRESST) at UCLA. Dr. Baker was responsible for a multi-year, 
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nationwide evaluation of the Apple Classroom of Tomorrow (ACOT) Program. After extensive 
analysis of classrooms and schools using Apple computers, she writes: 

Technology alone cannot improve teaching and learning. If it could, we would 
have documented improvements in student learning in all subject areas in all our 
technology-related evaluations. Technology use must be grounded firmly in 
curriculum goals, incorporated in sound instructional process, and deeply 
integrated with subject-matter content. Absent this grounding, which too often 
is neglected in the rush to glittery application, changes in student performance are 
unlikely.^ 

Dr. Baker is far from alone in this perspective. Papers presented at a recent national conference 
organized by the Secretary of Education contained the same message.^ Staff members from 
Education Development Center’s Center for Children and Technology wrote: 

. . . technologies by themselves have little scaleable or sustained impact on 
learning in schools. In order to be effective, innovative and robust technological 
resources must be used to support systematic changes in educational 
environments that take into account simultaneous changes in administrative 
procedures, curriculum, time and space constraints, school-community 
relationships, and a range of other logistical and social factors.^ 

The important point in these analyses is that technology - and its impact - cannot be 
considered apart from the context in which it is used. Technology is a tool that teachers and 
students can use to advance student learning. Different teachers and different students use it in 
different ways. Middle school students writing a history paper augment library resources by 
using the Internet. Elementary teachers use drill-and-practice programs to help students master 
multiplication facts. High school drafting students use computer aided design (CAD) programs 
to improve their rendering of buildings. In all of these examples, technology is a tool that 
supports learning. It is not the primary force that drives learning. 

Consider another technology we use on a daily basis - a kitchen stove. By itself, a stove 
adds nothing to our quality and flavor of our meals, even if we turn it on and it heats up. It is 
only when it is used in specific ways to prepare food that its contribution becomes important and 
apparent. How would we evaluate the stove’s contribution to the tastiness of prepared food? We 
would need to assess the entire context in which the food was prepared including the ingredients 
used, the manner and proportions in which ingredients were combined, the length of time they 
were cooked on the stove, and finally, and perhaps least importantly, the consistency of the 
stove’s heat. 

The situation is similar in classrooms where we are concerned about student learning 
outcomes instead of appetizing meals. Technology is, as one teacher told me, “just one part of 
the puzzle. You still have to consider what you and the student do with the technology, and how 
this fits into other instruction and assessment you have going.” I suggest that future evaluations 
of the Idaho Educational Technology Initiative emphasize the context, not merely the presence, 
of educational technology use. 
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The second chapter of this report discusses one model for context-sensitive technology 
evaluation in the research conducted by Al Strickland and Jack Coffland. Their mathematics 
research could be termed a “curriculum intervention” model. Strickland and Coffland developed 
a model of curriculum and instruction that included computer use, taught a group of teachers 
to implement the model, and examined the impact of this model on student learning. (Data 
collected on the students in comparison classrooms provided information about student learning 
in typical classrooms not currently implementing the Strickland-Coffland curriculum/ 
instructional model.) This research yielded not only information about the impact of a particular 
curriculum/instruction model, but provided insight into effective teaching and implementation 
processes. In the future, this evaluation approach could be enhanced by including additional 
process measures recording what actually went on in the model and comparison classrooms. 
This could be done by using observers or by including teacher and student self-report 
information. Attention to these classroom process variables, and classroom-level comparisons 
could yield information about model factors that have the most power to make a difference in 
student learning outcomes. 

Note that in such an evaluation, technology use is embedded in a curriculum and 
instructional context; it is not assumed that technology alone will have the major impact on 
student learning. Instead, technology is simply one of the tools being used to support and extend 
student learning. 

Use Existing Data Collection Mechanisms 

Cliff Green s dissertation research (summarized in “The Accountability Report”) made 
an important methodological contribution by demonstrating how classroom and student level 
data can be collected through the Idaho Statewide Assessment. This approach could be used to 
collect information about the context of technology use and the school subjects and learning 
goals being supported by computer technology. It could also provide a cost-effective way to 
canvass teachers about their hardware, software, support, and training needs. 



Mine the "Accountability" Data Set 
Collected by Cliff Green and Mike Rush 



Cliff Green s original data analyses provided initial evidence of the relationship between 
computer use and academic achievement gains, but did not use sophisticated statistical 
techniques that can take full advantage of the context information collected. Further analyses 
of the same data set using hierarchical linear modeling could yield information about the 
relationships among student socioeconomic status, school computer use, home computer use, 
type of computer use, type of school attended, and student achievement gain. To my knowledge, 
this data set is unique; no other data is available that links computer use with statewide 
achievement. I believe it would be well worth while to submit these data to more sophisticated 
statistical analysis. Knowledge gained in this analysis could be used to structure future statewide 
studies. 



Buck Institute for Education - Chapter 4 



4.3 



Use Disinterested, External Evaluators 



A final suggestion regarding future Idaho technology evaluations concerns the 
individual(s) charged with carrying out the evaluation. I believe that future evaluations can best 
be conducted by an experienced, impartial, external evaluator or an well-qualified evaluation 
organization (e.g.. Northwest Regional Educational Laboratory, Rockman, et. al., Stanford 
Research Institute, etc.) familiar with Idaho education. Idaho educators, be they employed by 
Idaho school districts or colleges of education, face difficult conflicts of interest, when asked to 
evaluate their own achievements. 
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12/26/95 


Gordon Fisher, Principal 


Legislative Services Office Interim 




Budget and Policy Analyst 


Evaluation of the Idaho Education 
Technology Initiative of 1994 



Purpose: 



Data sources: 



Conclusion: 



Recommendations: 



Summary (Report #1) 

This interim evaluation was mandated by Idaho Code §33-4807. 
It examined “the purposes of the [Idaho Educational 
Technology Initiative] legislation and the goals set forth in the 
statement of purpose. The evaluator also “solicited input from 
the members of the technology council.” 

ICTL minutes, grant applications, reports from school districts 
and other agencies, and interviews with ICTL members. 

That “the purposes of the legislation have been achieved and that 
several of the goals have been attained or progress made toward 
attainment ... the programs have not been in operation long 
enough to evaluate the relative impact, costs and benefits.** 

1) Funding for ICTL expenses be included in public school 
appropriations, and 2) Idaho Code §33-4807 be amended to 
provide for a comprehensive evaluation to be undertaken by the 
Office of the State Board of Education. 



Reviewer’s Comments: This early evaluation was conducted before ICTL goals played a 

prominent part in Initiative operations. Instead, it uses the goals 
set forth in the HB 901 Statement of Purpose as a focus. It uses 
equipment expenditure data and comments from ICTL members 
to document the fact that the Idaho Educational Technology 
Initiative was underway and was enthusiastically received by 
educators. 



Conclusions are cautious and sensible. Notably, it calls for a 
more comprehensive evaluation to examine the effective use of 
educational technology and its impact on students and its 
benefits for students and teachers. 
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Evaluation Assessment by lETI Goals 



Goal 


Methodological 

Rigor 


Generalizability 
to Idaho 
Teachers and 
Students 


Overall Quality 
of Research/ 
Evaluation 


Goal 1: Integration 








Technology Acquisition 


2 


2 


2 


Technology Use 


1 


2 


2 


Goal 3: Collaboration with 

Colleges of Education 


1 


2 


2 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be 
generalized beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 



BEST COPY AVAILABLE 
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1/22/96 


Idaho State Department of 


Interim Report of the Idaho Educational 




Education 


Technology Initiative of 1994 Funding for 
Public Schools 



Summary (Report #2) 

Purpose: This report examined the effect of the Idaho Educational 

Technology Initiative funding on technology in Idaho public 
schools. 

Data sources: Technology Grant Applications for 1994-1995; Technology 

Grant Applications and Progress Reports for 1995-1996; 

National Reports on Educational Technology; Legislative 
Services Offices Interim Evaluation; Transcripts from 1995 
Public Meetings on Technology; ICTL minutes; Conversations 
with Regional Technology Advisors, school district 
administrators and teachers. 

Conclusion: The largest single category of hardware being purchased is 

computers. The student to computer ration is 8.2:1. For the second 
year in a row, Idaho schools have a better student to computer ratio 
than the national average. Other technologies are being 
implemented at approximately the same rates as the national 
averages with the possible exception of school districts with satellite 
receivers. Teachers report that their greatest training need is in 
software selection and implementation. The second most frequently 
reported need is using the Internet. 

Recommendations: None made. 

Reviewer’s Comments: This report described the implementation of the Technology 

Initiative from May 1994 through June 1996. School district 
budget requests and reports appear to have provided the main 
source of information. Proposed expenditures by school districts 
were greater than Technology Initiative funds. Although other 
types of information were collected (see above) , it did little to 
inform the report’s conclusions. There was some disagreement 
between teachers and Regional Technology Advisors regarding 
the most pressing training needs. Steps were being made to 
provide teachers with needed technology training. 
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Evaluation Assessment by Applicable lETI Goals 



Goals 

Addressed 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Acquisition 2 

Technology Training 1 



2 

1 



2 

1 



Key to Ratings 





Methodological Rigor 




Generalizability 
to Idaho Teachers and 
Students 




Overall Quality 
of Research/ 
Evaluation 


0 


Serious methodological 
problems invalidate results 


0 


Results can not be 
generalized beyond current 
research/evaluation 


0 


Untrustworthy; conclusions 
should be ignored 


1 


Methodological problems 
compromise trustworthiness 
of results 


1 


Results can be generalized 
with caution 


1 


Flawed research/evaluation; 
consider conclusions 
skeptically 


2 


Acceptable methodology; 
trustworthy results 


2 


Results widely generalizable 


2 


Competent research/ 
evaluation; conclusions 
appear sound 


3 


Excellent methodology; 
compelling results 






'3 


Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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12/16/97 


Idaho State Board of 


State Board of Education Evaluation of the 




Education 


Idaho Education Technology Initiative for ■ 






FY 1997 



Summary (Report #3) 



Purpose: All educational agencies receiving state appropriate funds report 

to the SBOE on the impact, costs, and benefits of programs as 
relating the goals outlined in HB 901. The purpose of this report 
was to summarize the current status as it related to the above 
expectations. 

Data sources: Progress reports on school districts’ 1995 Technology Plans 

submitted to the State Department of Education. Reports 
emphasized revised budgets and progress made toward 
achieving the eight technology goals developed by the ICTL. 

Conclusion: The intent of the legislation has been attained or progress has been 

made toward that attainment . . . ICTLs ability to measure impacts 
or benefits in a consistent manner continues to be troublesome. 

Recommendations: ICTL should in partnership with the College of Education: 1) 

review the eight current goals of ICTL and emphasize those that 
are measurable; 2) develop statewide performance indicators 
based on the measurable goals; 3) use this performance data to 
track past progress documented in the annual reports for the FY 
1998 SBOE Evaluation; and 4) use this performance data to 
track future progress by making it part of future grant 
applications. 



Reviewer’s Comments: This evaluation emphasized ICTL goals and used them as a lens 

to organize thinking about what the Idaho Educational 
Technology Initiative should accomplish. Like previous 
evaluations, it highlights activities being undertaken with 
Technology Initiative funds and details expenditures according 
to budget categories. Information about the im.pact of these 
activities is anecdotal, and summarized from grant progress 
reports. As noted in the evaluation’s conclusion, evidence from 
unbiased, third-party observers would make a stronger case for 
significant educational impact. 
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Evaluation Assessment by Applicable lETI Goals 



Goals 


Methodological 


Generalizability 


Overall Quality 


Addressed 


Rigor 


to Idaho 


of Research/ 






Teachers and 


Evaluation 






Students 





Goal 1: Integration 








Technology Acquisition 


2 


2 


2 


Technology Training 


1 


1 


1 


Technology Use 


1 


1 


1 


Technology Impact 


1 


1 


1 


Goal 2: Compatibility 


1 


1 


1 


Goal 3: Collaboration with 

Colleges of Education 


1 


1 


1 


Goal 4: Community 
Collaboration 


1 


1 


1 


Goal 8: Systems Support 


1 


1 


1 


Key to Ratings 


Methodological Rigor 


Generalizability 
to Idaho Teachers and 
Students 




Overall Quality 
of Research/ 
Evaluation 


0 Serious methodological 0 

problems invalidate results 


Results can not be 
generalized beyond current 
research/evaluation 


0 


Untrustworthy; conclusions 
should be ignored 


1 Methodological problems 1 

compromise trustworthiness 

of rpci life 


Results can be generalized 
with caution 


1 


Flawed research/evaluation; 
consider conclusions 
skeptically 


2 Acceptable methodology; 2 

trustworthy results 


Results widely generalizable 


2 


Competent research/ 
evaluation; conclusions 
appear sound 


3 Excellent methodology; 
compelling results 




3 


Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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Debra J. Dirksen, Christine 
Bauer, David Cdffland, Sarah 
Naylor, Archie George 



Public School Technology Grant Program FY 
96 and FY 97 (Years 1, 2 & 3): A Statewide 
Evaluation of the Impact of Technology 
Integration on Idaho Teachers and Students 



Summary (Report #4) 

Purpose: This evaluation considers the first goal of the Idaho Educational 

Technology Initiative - Integration - and addresses 8 questions: 
1) How have districts used their technology funds? 2) What 
hardware and software have districts purchased? 3) What 
relevant training have teachers received; 4) What technology 
skills have teachers acquired? 5) How are teachers using 
technology? 6) What technology skills have students acquired? 

7) How have students used technology skills to problem solve, 
do school work, and address real-life situations? and 8) What 
impact has technology integration had on student performance? 

Data sources: State and University reports were used to answer the first 3 

questions. For the remaining 5 questions, a random sample of 
containing one or more schools from approximately 50% of 
Idaho school districts was drawn. Five teachers were randomly 
selected from each school completed a survey and were 
interviewed about technology use. 

Conclusions: 1) Technology is helping to foster positive learning behaviors and 

self-regulated learning strategies in students; 2) The quality of 
student work is improving through the integration of technology 
by teachers; 3) Interactive learning is being fostered in 
classrooms where technology is being integrated; 4) Technology is 
becoming a teaching and planning tool; 5) Students are 
developing technology skills associated with application software; 
6) Interaction between students and collaborative learning is 
being fostered in classrooms where technology is being 
integrated; 7) For many schools, technology has expanded the 
amount and range of resources available to them. For example, 
technology has enhanced many school libraries; 8) Technology 
allows teachers to focus on individual needs, as well as the needs 
of their entire class; 9) Through using technology, teachers are 
able to cover content areas in greater scope; 10) Teachers 
reported being able to use technology to address specific content 
issues as well as integrate content areas; 11) Technology is being 
used to develop both basic skills and problem solving skills; 12) 
Teachers at all levels reported finding methods for integrating 
technology within the curriculum to benefit their students; 13) 
Technology is being used for individualized instruction, whole 
group instruction, small group instruction, and collaborative 
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Recommendations: 



Reviewer s Comments: 



learning; 14) Technology is expanding the teachers potential. 
They are able to try new methods for teaching, organizational 
strategies, management strategies, and teaching tools; 15) E-mail 
is being used to facilitate communication between educators, 
parents and students. 

1) Teacher training for integrating technology is of primary 
importance; 2) computers should be a maximum of one 
generation behind business; 3) Schools need software so that 
they can more effectively integrate the technology available to 
them; 4) Funding is needed to provide on-site technicians to 
support the technology systems in place; 5) Financial support 
for increased access by libraries to on-line resources is necessary 
to capitalize on what is already taking place with general 
Internet access; 6)Technology funding needs to be increased, so 
teachers may more effectively integrate technology within the 
curriculum. 

This evaluation presents a good, broad-brush description of 
technology acquired by schools and the way it is used in the 
classroom. Less compelling are descriptions of teacher training 
and the impact of technology on teaching and learning, which 
relies solely on teacher perceptions. Some evidence is presented 
of collaboration with colleges of education and the community 
at large. In general, results appear applicable to Idaho educators 
as a whole. 

Links between data presented and conclusions drawn are 
sometimes tenuous, perhaps because a number of conclusions 
are drawn from interview data, and this data is not presented. 

The reviewer’s sense of the data is that the majority of Idaho 
teachers are competent using word processing, grade book 
programs, and where Internet access is easily available, using the 
Internet for research and email. Forty percent of students are 
competent with word processing software, and students 
frequently use technology to complete class projects. 

Future studies of this kind should focus more attention on grade 
level differences in pedagogy and technology use and 
supplement teacher perceptions with independently-assessed 
learning outcomes. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 








Technology Acquisition 


2 


2 


2 


Technology Training 


2 


2 


2 


Technology Use 


2 


2 


2 


Technology Impact 


1 


1 


1 


Goal 3: Collaboration with 

Colleges of Education 


1 


1 


1 


Goal 4: Community 
Collaboration 


1 


1 


1 



Key to Ratings 





Methodological Rigor 




Generalizability 
to Idaho Teachers and 
Students 




Overall Quality 
of Research/ 
Evaluation 


0 


Serious methodological 
problems invalidate results 


0 


Results can not be 
generalized beyond current 
research/evaluation 


0 


Untrustworthy; conclusions 
should be ignored 


1 


Methodological problems 
compromise trustworthiness 
of results 


1 


Results can be generalized 
with caution 


1 


Flawed research/evaluation; 
consider conclusions 
skeptically 


2 


Acceptable methodology; 
trustworthy results 


2 


Results widely generalizable 


2 


Competent research/ 
evaluation; conclusions 
appear sound 


3 


Excellent methodology; 
compelling results 






3 


Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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Idaho State Division of 


The Idaho Technology Initiative: An 




Vocational Education, Idaho 


Accountability Report to the Idaho 




State Department of 


Legislature on the Effects of Monies Spent 




Education, Bureau of 


Through the Idaho Council for Technology in 




Technology Services 


Learning (The ‘‘Accountability Report") 



Summary (Report #5) 

Purpose: Use the goals established by the Idaho Council for Technology 

in Learning (ICTL) for the Idaho Educational Technology 
Initiative as an organizer, this report presented multiple strands 
of research evidence bearing on the accomplishment of each 
goal. 

Data sources: Data included an analysis of the relationship of computer 

exposure and individual achievement gain scores in 
mathematics, reading and language conducted by Cliff Green, 
data collected in earlier evaluation efforts, and data collected by 
college of education staff and classroom teachers. 

Conclusions: 1) There is a positive relationship between academic performance in 

language math and reading and student technology exposure; 2) 

The defacto use of the Internet as a state network has resulted in 
technological compatibility among schools and districts; 3) Colleges 
of Education and School Districts have worked together to prepare 
teachers to use technology: 4) Community members have made 
investments in and have been involved in the implementation of 
technology in schools; 5) Schools are using technology to improve 
their administrative efficiency; 6) Students are being trained to 
install, maintain and support technology. 

Authors Conclusions: The benefits of technology in teaching and learning are clear: an 

increase in academic achievement in reading, mathematics, 
language and core studies, improved technology literacy, 
increased communication, well-trained, innovative teaching, 
oositive relationships with the community, more efficient 
operation of schools, and technically qualified students ready to 
enter today’s workforce. 

Reviewer s Comments: The report does a good job of bringing together multiple sources 

of data to provide information about the accomplishment of 
ICTL goals. Cliff Green’s study of the relationship of student 
achievement gains and technology exposure is ground breaking 
in its use of data from the Idaho Statewide Assessment. Further 
studies of this type, with improved methodology, can provide 
convincing information about the value added by technology 
use to student learning. 
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This report is a significant accomplishment, given the abstract 
nature of the ICTL goals, and the fact that no goal-driven data 
collection strategy was established at the time ICTL funding 
began. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Import 2 

Goal 2: Compatibility 2 

Goal 3: Collaboration with 2 

Colleges of Education 

Goal 4: Community 2 

Collaboration 

Goal 5: Technology Systems 2 

Goal 7: Student Training 2 



2 

1 

2 

2 

2 

2 



2 

2 

2 

2 

2 

2 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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G2 



4/98 


Clifford Green, Boise State 


The Relationship Between Technology 




University &. Idaho State. 


Exposure and Academic Improvement in 




Department of Education 


Idaho’s K- 12 Public Schools 



Summary (Report #6) 

Purpose: This dissertation examined whether students who have more 

exposure to computers receive higher achievement test scores 
than students with less exposure. 

Data sources: Student self-report data on computer exposure was collected 

from eighth- and eleventh-grade students as part of the Idaho 
Statewide Assessment in 1998. In addition, individual 
achievement gain scores in mathematics, reading and language 
were calculated using 1994 and 1998 Idaho Statewide 
Assessment results. Using these two data sets, the relationship 
between achievement gain and computer exposure was 
examined. 

Conclusions: 1 ) Eighth grade students who reported high exposure to technology 

experienced additional gains of 2.7 months in math, 2.1 months in 
reading, and 1 month in language compared to students who had 
low exposure to technology; 2) Eleventh grade students who 
reported high exposure to technology experienced additional gains of 
1 month for math and language, and less than a month for reading 
compared to students who had low exposure to technology; 3) After 
controlling for socioeconomic status, smaller additional gains of 1.4 
months in math, 1.1 months in reading, and 1.4 months in language 
were reported for the eighth grade group and 1.3 months in reading 
were reported for the eleventh grade group.; 4) Of all types of 
technology exposure, the ability of the student to choose the 
appropriate software tool had the greatest relationship with 
achievement gain. 

Authors Conclusions: 1) A longitudinal study be conducted over the next 4 years to 

verify and strengthen results of this study: 2) A follow-up study 
should identify school buildings and/or districts which scored 
high on the statewide assessment to gather information 
regarding teacher training, hardware implementation, and 
Internet strategies and policies; 3) Follow-up studies considering 
the relationship to subject-specific achievement gains and 
technology exposure should be conducted; 4) Follow-up studies 
using a more rigorous control for student socioeconomic status 
and level of computer use should be conducted. 

Reviewer s Comments: The study is to be commended as an initial effort and is 

significant as a demonstration of the type of technology research 
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BEST COPY AVAILABLE 



that can be conducted at little cost to teachers, students, or the 
state using the Idaho Statewide Assessment. Although the author 
attempts to control for student socioeconomic status, the 
measure used (home computer use) is inadequate to rule out the 
competing explanation that students who frequently use 
computers in school attend schools enrolling higher income 
students and which have better educational environments than 
schools where students do not use computers frequently. 

Further analyses of the existing data set using hierarchical linear 
modeling do a more powerful job of isolating achievement 
effects due mainly to socioeconomic status from those associated 
with school attendance and computer use. This would provide 
more compelling information about the impact of computer use 
on tested student achievement. 
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No Date 



Audrey C. Rule & Manuel T. 
Barrera, Boise State University 



Effects of Technology and Curriculum 
Integration on 3"'"^ grade Students' 
Technology, Thinking, and Science Process 
Skills (Research Summary) 



Summary (Report#?) 



Purpose: This study evaluates how three different curriculum integration 

approaches enhance learning in science content acquisition, 
vocabulary development, and computer technology proficiency. 

Data sources: Three third-grade classes were randomly assigned to one of 

three different curriculum approaches: 1) thematic instruction, 

2) project-based instruction, and 3) use of curriculum-based 
object boxes. Students also used database, spreadsheet, and word 
processing software to develop products associated with each 
approach. Pre- and post-tests were conducted prior to and 
following the instructional units. Data included assessments of 
content knowledge, questioning skills, observation skills, 
descriptive vocabulary skills, classification skills, and technology 
competency skills as well as demographic and standardized 
achievement data. 



Results: 1) All students improved in technology self-efficacy and objective 

knowledge; 2) Students in the theme-based approach 
demonstrated significantly higher objective knowledge growth 
than the other two groups; 3) All three classes showed growth in 
applied descriptive vocabulary. 

Conclusions: No conclusions drawn. 



Reviewer s Comments: The research would be more useful if there was more 

information about what actually happened in each class; this 
would provide a basis for generalization to other teachers and 
classes. Still, the design appears adequate and the results 
demonstrate that students can learn about computer technology 
at the same time they are learning about subject matter. 
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0 Serious methodological 
problems invalidate results 
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compromise trustworthiness 
of results 
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0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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Del Siegle, Theresa Foster & 


The Impact of Presentation Software on 




Bruce Bedell, Boise State. 


Secondary Science Students' Achievement and 




University 


Attitudes (Research Summary) 



Summary (Report #8) 



Purpose: 



Does exposure to multimedia and presentation software increase 
student achievement in science courses? 



Data sources: Two high school anatomy and physiology classes participated in 

the research. The first class used laptop computers with 
presentation, multimedia and word processing software during 
the first semester of the 1997-1998 school year. The second class 
was taught using traditional means. During the second semester, 
the second class used the laptops while the first class was taught 
with traditional means. Both classes were taught by the same 
teacher using the same curriculum. Student achievement was 
measured each quarter with teacher-made tests. 

Results: Although the traditionally instructed class initially scored higher 

than laptop group, once students in the other class began using 
the laptops this difference decreased and eventually disappeared. 
When the semester changed and the new class was given laptops, 
their achievement was again higher then the class now receiving 
traditional instruction. 

Conclusions: This study clearly demonstrates that exposure to multimedia 

and presentation software increases student achievement in a 
high-level science course such as anatomy and physiology. 



Reviewer’s Comments: Although the research design could have benefitted from the 

inclusion of a pre-test for both classes, the design appears 
adequate and the results make a compelling case for the 
potential of computer technology to have an impact on student 
learning. It is difficult to generalize the instructional approach 
used in this study without more information about the manner 
in which computer technology was used in the classroom. 
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Goal 
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Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 2 



2 



Key to Ratings 



Methodological Rigor Generalizability Overall Quality 

to Idaho Teachers and of Research/ 

Students Evaluation 



0 Serious methodological 0 

problems invalidate results 



1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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Dawn Stram Statham & Clark 


Computers in the Classroom: The Impact of 




R. Torell, Boise State 
University 


Technology on Student Learning 



Summary (Report #9) 

Purpose: This document summarized research reports on the impact of 

technology on student learning. 

Data sources: Thirty-eight research reports appearing between 1990 and 1995 

as well as 10 compilations of studies (meta-analyses) appearing 
between 1983 and 1995. 

Results: 1) On their own, computers do not improve student learning. 2) 

When used appropriately in combination with targeted curriculum, 
instruction, and assessment practices, computer technology has been 
consistently shown to make a difference in student learning. 3) 

There are relatively few evaluations in the educational literature 
that are based on empirical data rather than impressionistic reports. 
4) Computers appear to be an especially effective instructional 
methodology with at-risk students. 5) Meta-analyses provide 
impressive evidence for the impact of computer-based instruction on 
student achievement. 

Authors’ Conclusions: 1) Teachers need more training in the effective integration of 

computers into instructional practice. 2) Proper implementation 
of computer technology in schools requires convenient and 
equitable computer access, curricular integration, and attention 
to student learning. 3) More research is needed to evaluate the 
critical practices and contexts that enable computer-enhanced 
instruction to make a difference in student achievement. 

Reviewer’s Comments: Well-conceived and organized report. Documents that although 

computers can make an important difference in student learning 
and motivation, the school and instructional context in which 
they are used is crucial. Just as a new set of cookware will not 
automatically improve the chefs cooking, the acquisition of 
computer technology will not automatically improve learning. 
Like cooking, it is the ingredients and the way they are 
combined that make the difference in student learning. 
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Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 3 2 



3 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation: conclusions have 
import for Idaho education 
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4/99 



Deborah E; McGrath, Boise 
State University & Kuna, 
School District. 



The Relationship Between Computer Use, 
Student Academic Achievement, and Student 
Perceived Technology Competency 



Summary (Report #io) 

Purpose: This masters thesis examined the relationship between student 

home computer use and gains on standardized achievement tests 
in reading, math, language and with student technology 
competency. 

Data sources: Approximately 12, 000 Idaho 8^’^ grade students and 12,000 

grade students completed a Technology Exposure Survey in fall 
1998. These students had attended Idaho public schools and had 
completed standardized achievement tests for reading, math, 
and language during the 1994-95 school year. 

Results: 1) There was a small but statistically significant positive 

correlation between home use and achievement in reading, language 
and math, and a strong positive correlation between home computer 
use and perceived technology competency; 2) In both comparisons, 
the 8^^ grade group showed stronger correlations than the IV^' grade 
group. 

Authors’ Conclusions: 1) The positive correlations, though small, are significant and 

deserve further investigation; 2) Computers are a part of life 
today and students need access to computers outside of school 
time; 3) Socioeconomic factors that are related to computer 
ownership and availability need to be considered in future 
research; 4) The instruments used in the study need to be 
refined; 5) Future research with additional data collection 
strategies should be conducted yearly, rather than at a four-year 
interval. Comparisons in computer use should be made at the 
classroom level. 

Reviewer’s Comments: The most important contribution this study makes is 

methodological. The use of the Idaho statewide assessment 
program to collect information about computer use (and other 
classroom variables) is a time- and cost-effective data collection 
strategy. As was mentioned in the review of Cliff Green’s 
dissertation research using the same data set, further analysis 
using more powerful statistical techniques can tease out the 
relationship of school, home, computer usage, and 
socioeconomic factors. 
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import for Idaho education 
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ND 


Del Siegle and 'Theresa Foster, 


The Status of Vallivue Consortium Teacher 




Boise State University . . 


Attitudes Toward Technology in the 






Classroom 



Summary (Report #io) 

Purpose: This research examined the attitudes of Southwest Idaho 

teachers towards integrating computers within their classroom 
instruction. 

Data sources: Seven hundred eighteen teachers from 10 school districts 

completed a computer attitude survey in August, September and 
October of 1997. 

Results: 1 ) Approximately 60% of teachers were incorporating computers 

into their classroom instruction. 2) Approximately 70% of 
teachers reported they needed to improve their computer skills 
and 80% expressed that they wanted to learn more about using 
computers in the classroom. 3) Approximately 60% of teachers 
believed that they could master computer skills without undue 
effort. 4) There was general agreement that computers will 
improve education, and make their instruction more productive, 
efficient and creative. 5) There was general agreement that 
computers help students learn more efficiently, increase student 
interest and motivation, and make students better prepared for 
the workforce. 

Authors' Conclusions: 1) Vallivue Consortium teachers clearly believe that including 

computers in their classroom instruction increases student 
learning and provides additional educational opportunities for 
students. 2) Vallivue Consortium teachers are eager to receive 
additional technology training. 

Reviewer’s Comments: Useful portrait of the attitudes of a group of Idaho teachers. 

Authors could have generated additional, and potentially useful, 
information by examining differences among Leadiers at 
different grade levels, subjects taught, or in different district 
contexts. Small sample limits generalizability. 
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No Date 



Manuel.T! Barrera, III, Audrey 
C. Rule, G. Jolene Dockstader, 
& John A.-Derr, Boise State 
University 



Comparing Technology Skill Development in 
Computer Lab versus Classrooms Settings of 
Two Sixth Grade Classes 



Summary (Report #ii) 

Purpose: This research examined whether students learned computer 

skills more effectively in a classroom or in a computer lab. 

Data sources: Forty-three mixed ability six grade students from two intact 

classrooms were instructed in the basic computer skills of file 
management, word processing, and Internet access. Students 
completed performance-based pre- and post- computer skill 
tests. Students also completed an attitude survey about 
computer use at the end of the instruction period. 

Results: 1) Students learned computer skills more effectively in the 

computer lab setting. 2) Attitudes toward computer use were 
equivalent in both instructional settings. 

Authors’ Conclusions: 1) The context of regularly scheduled computer lab sessions 

provides more time for all students to master computer skills 
than the regular classroom setting. 2) Computer time is 
monopolized in the classroom setting by higher achieving 
students and those who particularly enjoy using the computer. 

3) Computer lab instruction is more intensely focused on 
computer skill learning and less subject to classroom and 
student distractions than the classroom. 4) Instructional setting 
does not affect student attitudes toward computer use. 

Reviewer’s Comments: Competently-designed study which illustrates well the impact of 

the social context of classrooms on computer use and the 
development of computer skills. Raises the important question 
of strategies classroom teachers can use to ensure equal access 
and use of classroom computers. 
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evaluation; conclusions have 
import for Idaho education 



A- 32 



Buck Institute for Education- Appendix A 



7/99 



Jennifer Jarvis, Robert Rogers, 
Julie Oerke,: Anne Lane, &. 
Kay Piva, Challis High School, 
Challis Joint School District 



Is technology more effective in the 
memorization of multiplication facts than 
traditional methods? 



Summary (Report #12) 



Purpose: This action research compared fifth-grade students who used a 

software program with other students who used paper flash 
cards to learn multiplication facts. 



Data sources: 



Twenty-three fifth-grade students completed pre- and post-tests 
on multiplication facts. 



Results: 



There was no difference in score changes between students who 
used the software and students who used paper flash cards. 



Authors’ Conclusions: 



1) Future research should be conducted with larger numbers of 
students. 



Reviewer’s Comments: This action research study is useful because it demonstrates that 

although computer technology opens up many opportunities for 
classroom integration, technology will not always prove to be 
the instructional method of choice. 
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Samuel Wadsworth; North 


Can the use of drill and practice software 




Gem Elementary, North Gem 


take the place of written practice in 




School District 


grammar? 



Summary (Report #13) 



Purpose: This action research examined whether students who used 

“School House Rock” software to supplement the textbook 
learned grammar more effectively. 

Data sources: Eleven third-grade students were divided into two groups. Both 

groups received the same grammar lessons on adverbs. 
Following the lesson, one group completed the exercises in the 
textbook. The other group used the computer to practice. 



Results: Both groups demonstrated roughly equivalent gains from pre-test 

to post-test 

Authors’ Conclusions: 1) Although it appears that grammar software may be capable of 

replacing written work, the results of this study are equivocal 
because of the small number of students who participated. 



Reviewer’s Comments: The author is to be commended for controlling the practice time 

of the computer and textbook groups, and for not going beyond 
the results presented by the data. 
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1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
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2 Competent research/ 
evaluation; conclusions 
appear sound 
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evaluation; conclusions have 
import for Idaho education 
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7/99 


Cinci Canine, Gooding Middle 


Does the Skillbuildine Kevboardine Software 




School, Gooding Joint School 


Help Students Increase Their Words Per 




District 


Minute and Typing Accuracy? 



Summary (Report #i 4) 



Purpose: This action research examined the usefulness of a software 

program in teaching typing to seventh- and eighth-grade 
students. 



Data sources: Forty-four seventh- and eighth-grade students were assessed for 

typing speed and accuracy at the beginning and end of the 
program. 

Results: Most students demonstrated gains in keyboarding speed and 

accuracy. 

Authors* Conclusions: 1) The S/ciJJbuiJding program was effective in increasing students’ 

typing speed and accuracy. 2) Further research should compare 
the impact of the Skillbuilding software with other programs that 
teach keyboarding, and the success of Skillbuilding with different 
student populations 



Reviewer’s Comments: This is useful action research that demonstrates the utility of 

integrating educational software within a program of classroom 
instruction. It is unclear whether the software is more effective 
than traditional instructional methods. 
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of Research/ 
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problems invalidate results 

1 Methodological problems 
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of results 

2 Acceptable methodology; 
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3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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Afton Diemart, Jefferson 
Elementary School, Jerorne 
Joint School District 



Can First Graders Learn Keyboarding Skills? 



Summary (Report #15) 



Purpose: 



This action research examined whether or not first-grade 
students can learn keyboarding skills. 



Data sources: 



Eighteen first-grade students. 



Results: Students typed an average of 11 words per minute with 84% 

accuracy 

Authors’ Conclusions: The majority of students were capable of learning proper 

keyboarding skills. 



Reviewer’s Comments: 



Useful action research that demonstrates younger students’ 
ability to keyboard. 
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of Research/ 
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of Research/ 
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0 Serious methodological 
problems invalidate results 

1 Methodological problems 
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of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology: 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy: conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation: conclusions have 
import for Idaho education 
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7/99 


Christine Hooyer. Mackay 


Will Third Grade Students Improve 




Elementary School, Mackay 


Multwlication Skills Usins Speed Math 




Joint School pistrict 


Software Better than They Would Using 
Traditional Instructional Methods? 



Summary (Report #16) 



Purpose: This action research examined the usefulness of a software 

program in teaching basic multiplication skills to third-grade 
students. 



Data sources: 



Twelve students took a paper and pencil pre-test and post-test 
on multiplication facts. 



Results: There was an overall improvement in students' performance in 

both time and computational accuracy. 

Authors’ Conclusions: 1) The software helped students to learn basic multiplication 

facts. 2) Further research with a control group and a larger 
population of students is needed. 



Reviewer’s Comments: Useful action research that demonstrates the utility of 

integrating educational software within a program of classroom 
instruction. 
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2 Results widely generalizable 2 Competent research/ 

evaluation; conclusions 
appear sound 


3 Excellent methodology; 
compelling results 


3 Exemplary research/ 

evaluation; conclusions have 
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Janice Clark; Filer Elementary Can Identification of Rhyming Words be 
School, Filer School District Improved by Using Computer Technology for 

Drill and Practice? 



Summary (Report #17) 



Purpose: This action research tracked the growth of third-grade students’ 

reading during the 1998-99 school year. 

Data sources: Eighteen grade students completed pre- and post-tests on 

their understanding of rhyming words. 



Results: 



The average student score increased between pre-test and post- 
test 



Authors’ Conclusions: 1) Reader Rabbit 2 is a useful way to teach rhyming to students. 

2) Future research should be conducted with a larger, more 
diverse population and include a control group that learned 
about rhyming in a traditional manner. 

Reviewer’s Comments: Useful action research that demonstrates the utility of 

integrating educational software within a program of classroom 
instruction. Unclear whether the software is more effective 
than traditional instructional methods. 
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of Research/ 
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of Research/ 
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1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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7/99 


Gay la Coombs, A. W. Johnson 


Will Mammoth Meltdown help students learn 




Elementary School, Firth 
School District 


adverbs and adjectives? 



Summary (Report #18) 



Purpose: 



This action research tracked the growth of third-grade students’ 
reading during the 1998-99 school year. 



Data sources: 



Twenty-three third-grade students completed pre- and post-tests 
on their understanding of adverbs and adjectives. 



Results: 



The average student score increased between pre-test and post- 
test. 



Authors’ Conclusions: 1) The degree of improvement shown has occurred in previous 

years without the use of software. Therefore, the researcher felt 
that the software did not have a positive impact on student 
learning. 2) More structured data collection routines would 
improve further research, as would the inclusion of a control 
group. 

Reviewer’s Comments: I applaud the researcher’s analysis and skeptical stance. As she 

indicates, without a control group, it is impossible to isolate the 
role the software in increasing students’ reading skills. 
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Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 2 1 



2 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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Bev Crawford, Mackay 


Will How the West Was One + Two x Four 




Elementary School, Mackay 


Help Second Grade Students Master Basic 




Joint School District 


Order of Operations in Mathematics? 



Summary (Report #19) 



Purpose: This action research examined the usefulness of a software 

program in teaching the order of operations to second-grade 
students. 



Data sources: Sixteen students took a pre-test and a post-test on order of 

operations. 

Results: Students demonstrated substantial gains in demonstrating 

competency with order of operations. 

Authors’ Conclusions: 1) The software was an important part of the instructional 

approach which also included lecture, demonstration and 
practice with paper and pencil. 2) Further research is needed 
using a control group design to isolate the impact of the 
software itself on student learning. 

Reviewer’s Comments: Useful action research that demonstrates the utility of 

integrating educational software within a program of classroom 
instruction. Unclear whether the software is more effective 
than traditional instructional methods. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 2 



2 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy: conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation: conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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6/9/99 


Albert W. . Strickland Jack . 


Mathematics Technology Project 




A. Coffland, Idaho State . 
University . 





Summary (Report #20) 

Purpose: This study examined whether students* mathematics 

achievement could be raised using an instructional approach 
combining: 1) precision mathematics teaching; 2) concrete, 
hands-on teaching aids; and 3) computer technology to assist 
the practice and application of math skills. 

Data sources: Five hundred and sixty 4^^ grade students in classes taught by 29 

teachers took part in the month project. Two hundred ten 
students participated as a control group. The Iowa Test of Basic 
Skills and the Stanford Diagnostic Mathematics Tests were used 
to assess change in students’ mathematics achievement. 

Results: Although there was significant variation in class-level change 

scores, on average, students who participated in the Mathematics 
Technology Project demonstrated considerably higher gains than 
students in the control group. 

Authors’ Conclusions: 1) Fourth graders’ mathematics achievement scores can be 

improved within an academic year; 2) The level of improvement 
depends upon the commitment of the school system, the 
building principal, and the classroom teacher; 3) The 
instructional model was as important as the use of technology in 
accounting for student gains; 4) Teachers who embraced both of 
these components were the most successful in raising student 
achievement. Although it appears that grammar software may be 
capable of replacing written work, the results of this study are 
equivocal because of the small number of students who 
participated. 

Reviewer's Comments: This is a well-designed study demonstrating that the classroom 

use of computer technology, by itself, does not lead to increased 
student achievement. Instead, it is the instructional context in 
which technology is used that makes the difference. Although it 
will not affect the study’s conclusion or its importance, it would 
be preferable methodologically to use NCE scores rather than 
percentile ranks in the statistical analyses, and to conduct the 
Analyses of Variance at both the classroom as well as the 
population level. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



'Goall: Integration 

Technology Impact 2 



2 
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Key to Ratings 





Methodological Rigor 




Generalizability 




Overall Quality 








to Idaho Teachers and 




of Research/ 








Students 




Evaluation 


0 


Serious methodological 


0 


Results can not be generalized 


0 


Untrustworthy; conclusions 




problems invalidate results 




beyond current 
research/evaluation 




should be ignored 


1 


Methodological problems 


1 


Results can be generalized 


1 


Flawed research/evaluation; 




compromise trustworthiness 




with caution 




consider conclusions 




of results 








skeptically 


2 


Acceptable methodology; 


2 


Results widely generalizable 


2 


Competent research/ 




trustworthy results 








evaluation; conclusions 
appear sound 


3 


Excellent methodology; 






3 


Exemplary research/ 




compelling results 








evaluation; conclusions have 
import for Idaho education 
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John Davis, . Michael Odell, 


An Evaluation of Traditional and Internet 




University of Idaho-Mdscow 


Approaches to Teaching Teachers about 






Educational Technology (Research 






Summary) 



Summary (Report # 21 ) 



Purpose: 



Data sources: 



Results: 



Conclusions: 



Is there a statistically significant difference between a group 
taught via the Internet, a group taught traditionally, and a group 
integrating traditional instruction with Internet enhancement? 

Fifty-one teachers participated in three treatments (traditional, 
Internet only, and traditional + Internet). Each group was 
exposed to 12 modules covering different types of educational 
technology, and completed pre- and post-tests of their 
knowledge.. 

There was no statistically signiRcant difference between the 
groups; the Internet group performed as well as both the 
traditional and integrated groups. 

Carefully developed and implemented Internet instruction is a 
viable, alternative instructional delivery mode (contingent upon 
content and course design) . 



Reviewer’s Comments: Although details are sketchy, the study appears to be 

satisfactorily designed. The use three groups is a nice design 
touch. It demonstrates that in an appropriate context Internet 
developed instruction can be used instead or in combination 
with traditional instructional approaches. 
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Go3l 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 2 



2 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0. Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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4/5/96 


Jerome Reinger, University of 
Idaho - 


A Longitudinal Study of Third Grade 
Achievement Test Scores Over a Three-Year 
Period at a Year-Round School of choice 
Using the Teaching and Learning with 
Computers (TLC) Method of Instruction 
(Dissertation) 






Summary (Report #22) 

Purpose: This dissertation investigated the use of Computer-Assisted 

Instruction in a year-round magnet school to determine the 
effects of these variables on student achievement in vocabulary, 
reading, language, math, social studies, and science. 

Data sources: Standardized achievement test scores were collected in the third, 

fourth, and fifth grades for 81 students (44 girls and 37 boys). 

Results: 1 ) Students' vocabulary, language, and reading scores were greater 

than predicted; there was no difference for mathematics, social 
studies or science; 2) These findings were true for both boys and 
girls; 3) The attendance track in which a student was placed was 
related to differences in student achievement in language and 
mathematics. 

Authors' Conclusions: 1) The results of this study should not be generalized beyond 

this particular group of students; 2) Evidence is provided that a 
combination of technology, parental choice, and year-round 
education may increase student achievement; 3) The relatively 
low level of students’ mathematics achievement suggests the 
need for a more effective mathematics curriculum; 4) Technol- 
ogy by itself is not responsible for gains in student achievement; 
an effective curriculum combined with competent teachers and 
strong parental support are necessary to produce a superior 
educational environment; 5) Further research is needed to 
replicate the findings of the study. 

Reviewer s Comments: An interesting documentation of the importance of combining 

technology with curriculum. The research design would be 
strengthened by the inclusion of a control group. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological 

Rigor 



Generalizability 
to Idaho 
Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 2 1 



2 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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Shirley M. Bryant, Boise State 


The Effect of Word Processing on Fourth 




University 


Graders’ Writing 



Summary (Report #23) 

Purpose: This research examined the impact on using word processors on 

students’ composition skills. 

Study Design Forty-seven fourth-grade students in two classes participated in 

the study. Both groups completed an initial writing assignment 
using paper and pencil. Both groups received 100 minutes of 
writing instruction each week for six weeks. The treatment 
group were taught to use word processors to create and edit 
their work. The contrast group continued to use paper and 
pencil. A final writing assignment was completed. The pre- and 
post-writing assignments were scored using the Idaho Fourth 
Grade Direct Writing Assessment grading standards. 

Results: After controlling for initial differences in writing skills, a 

comparison of pre- and post-writing scores indicated that students 
who had used word processors had become better writers. 

Authors’ Conclusions: 1) Instruction in word processing improves student writing, but 

students need to be taught how to use word processors within 
the writing/editing process. In other words, it is not the simply 
using word processors that improves writing, but knowing how 
to use them for writing improvement. 2) Keyboarding skills 
should be taught at the third grade or earlier. 3) Computers 
should be accessible to all children in every school. 

Reviewer’s Comments: Serious methodological problems raise serious questions about 

validity of results. Since students received writing instruction 
from two different teachers, it is impossible to know whether 
differences in writing accomplishment were attributable to 
differences in the teachers’ writing instruction or to the use of 
word processors. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 



0 



0 



0 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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No Date 


Pat Pinkerton, Del Siegle, 


A Comparative Study of Geometry 




Kelley Davies & Steve Dyke, 


Achievement using Geometer’s Sketchpad 




Boise State University 


(Research Summary) 



Summary (Report #24) 

Purpose: The purpose of this study was to determine if students who 

study geometric principals using the Geometer’s Sketchpad 
program demonstrate more in-depth understanding than those 
students who do not use the program. 

Data sources: Two geometry classes are using Geometer’s Sketchpad as part of 

their geometry instruction. Two other classes are receiving 
traditional instruction. 

Results: 1) There was no significant difference in geometry achievement 

between the classes using Geometers Sketchpad software and the 
classes that received traditional instruction; 2) Students' prior 
achievement was predictive of geometry achievement 

Conclusions: No conclusions drawn. 

Reviewer’s Comments: The research summary leaves out important details of how the 

research was conducted and make it difficult to assess the 
validity of the research. The results provided indicate that the 
inclusion of a technological tool in an instructional program 
does not necessarily have an impact on student achievement. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 


Methodological Generalizability Overall Quality 
Rigor to Idaho of Research/ 

Teachers and Evaluation 

Students 


Goal 1: Integration 




Technology Impact 


1 0 1 





Key to Ratings 


Methodological Rigor 


Generalizability Overall Quality 

to Idaho, Teachers and of Research/ 

Students Evaluation 


0 Serious methodological 
problems invalidate results 


0 Results can not be generalized 0 Untrustworthy; conclusions 
beyond current should be ignored 

research/evaluation 


1 Methodological problems 
compromise trustworthiness 
of results 


1 Results can be generalized 1 Flawed research/evaluation; 

with caution consider conclusions 

skeptically 


2 Acceptable methodology; 
trustworthy results 


2 Results widely generalizable 2 Competent research/ 

evaluation; conclusions 
appear sound 


3 Excellent methodology; 
compelling results 


3 Exemplary research/ 

evaluation; conclusions have 
import for Idaho education 
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Audrey C. Rule and Manuel T!. 


The Effect of Student Use of Text versus 




Barrera, III, Boise State 


Internet Resources on Achievement in a 




University 


Sophomore High School Biology Class 



Purpose: 

Data sources: 



Results: 



Authors’ Conclusions: 



Reviewer’s Comments: 



Summary (Report #25) 

This research examined the impact on student achievement in 
Biology of the use of text-based and Internet-based resources. 

Fifty-five mixed-ability sophomore students completed units on 
mollusks and annelids. At different times, each student 
experienced both the Internet-based and text-based condition. 
Pre- and post-treatment achievement data were collected using 
teacher-designed unit tests. 

1) For one of the units, students learned more using the Internet- 
based resources; there was no difference in learning for the other 
unit. 2) Students preferred using Internet-based resources over 
text-based resources. 

1) Internet resources should be pre-identified and student use 
structured to maximize learning. 2) Given that there was a 
positive increase in learning with one unit, and no decrease in 
learning with the other unit, teaching with Internet resources 
(in contrast to textbook and other text-based resources) is seen 
as a viable alternative to text-based instruction. 

Competently-designed study, although it is impossible to tell 
from the description of the Internet-based and text-based classes 
whether the material covered and the nature of the study guides 
students completed were equivalent. Small number of students 
and classes limit the generalizations that can be made from this 
study. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 



I 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology: 
trustworthy results 

3 Excellent methodology: 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy: conclusions 
should be ignored 

1 Flawed research/evaluation: 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation: conclusions 
appear sound 

3 Exemplary research/ 
evaluation: conclusions have 
import for Idaho education 
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Carolyn Thorsen, Richard;. 


The Effects of Hypermedia A ugmented 




Johnson, Jolene Dockstader, 


Lessons on Learning in a Sixth-Grade Social 




& Teri Romshek, Boise. State 
University 


Studies Class 



Summary (Report #26) 



Purpose: This research examined whether teacher-developed hypermedia 

learning materials have a positive impact on student social 
studies achievement. 



Data sources: Forty-five sixth-grade students divided into treatment and 

comparison groups. A 30-item test covering content knowledge, 
thinking skills and geographic knowledge was administered pre 
and post. 

Results: 1) There was no difference in the achievement of students in the 

treatment and comparison groups on the content and thinking 
skills part of the test. There was a difference, however, in 
students geographic knowledge. Students using the hypermedia 
materials demonstrated greater geography knowledge than those 
students using traditional materials. 2) Students in the 
hypermedia condition demonstrated more teamwork and 
developed higher quality answers than students in the comparison 
group. 



Authors' Conclusions: 1) Students using teacher-developed hypermedia learning 

materials learned content as well as or better than students using 
traditional classroom learning materials. 2) The effectiveness of 
the materials provides evidence that teachers should learn to 
prepare such hypermedia materials to enhance student learning. 

Reviewer s Comments: The finding of “no difference” is important because it frees 

teachers to use computer technology in innovative ways. Small 
sample limits strength of conclusions and generalizability. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological’ Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 1 0 



1 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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. Pam West, Ricmard M, 


The Effects of Integration of Text Reading 




Johnson, & Carolyn D. 


software on the Reading Level of Fifth-grade 




Thorsen, Boise State 


Students 




University 





Summary (Report #27) 



Purpose: 



This research examined the impact of text-reading computers on 
fifth-grade students’ reading fluency. 



Data sources: Twenty-nine fifth-grade students, including nine special 

education students, participated in the research. Their reading 
skill was assessed in the fall, winter, and spring, using a 
curriculum-based reading test, and the STAR computer-based 
reading assessment in fall and spring. 

Results: 1) Both reading measures indicated that the reading skills of 

regular and special education students increased over the course 
of the year. 

Authors* Conclusions: 1) The text reading software played an important role in reading 

instruction, 2) All classrooms should have this technology 
available. 



Reviewer s Comments: Poorly-designed study. Without a comparison group it is 

impossible to know if the text-reading software was really 
responsible for reading gains. It is unfortunate the design is 
faulty, as the basic idea - that computerized text reading can 
help students improve their own reading - is intriguing and 
important. 
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Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 0 0 0 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 



0 



Results can not be generalized 
beyond current 
research/evaluation 



0 Untrustworthy; conclusions 
should be ignored 



1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 



1 Results can be generalized 
with caution 

2 Results widely generalizable 



1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 



3 Excellent methodology; 
compelling results 



3 Exemplary research/ 

evaluation; conclusions have 
import for Idaho education 
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No Date 


Independent School, DiiStrict of: 


Waterford Early Reading Pilot Program 




Boise City ' - 


Evaluation (Research Summary) 



Summary (Report #28) 



Purpose: 



Data sources: 



Results: 



Conclusions: 



This evaluation was undertaken to assess the effectiveness of the 
Waterford Early Reading Program. 

Kindergarten students at three schools completed the Waterford 
Early Reading Inventory as a pre-test. The Waterford program 
was then implemented in two of the three schools. Students at 
the third school served as a control group. All students 
completed the Waterford Early Reading Inventory at the end of 
the year. 

Students receiving the Waterford program outperformed the 
control group by a significant margin (p < ,06) 

The Waterford program has been implemented in all 34 district 
elementary schools. 



Reviewer s Comments: Although details are sketchy, the study appears to be 

satisfactorily designed. The use of an outcome measure supplied 
by the software manufacturer raises some questions. It is 
possible the students in the Waterford program scored higher 
than the control students because the outcome measure more 
closely matched what they had been taught. A more convincing 
demonstration would employ a “neutral” assessment 
instrument. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 1 



1 



1 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology: 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research7evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation: conclusions have 
import for Idaho education 



BEST COPY AVAILABLE 



A- 66 



Buck Institute for Education- Appendix A 




111 



7/99 Shelly Christensen, Filer High Comparison of Web Site Creation Software 

. School, Filer School District 





Summary (Report #29) 


Purpose: 


This action research compared the new web site and image 
development software developed by Macromedia with software 
currently being used at the high school. 


Data sources: 


Ratings by students in a high school computer productions 
course. 


Results: 


1) Currently used software received higher ratings than the 
Macromedia software with the exception of Flash 2) Computers 
installed in the school were not powerful enough to run 
Macromedia software. 


Authors' Conclusions: 


1) Hardware capabilities should be considered before evaluating 
software. 2) Similar studies should be undertaken with future 
classes. 


Reviewer’s Comments: 


Research report is hard to follow and contradicts itself. Data 
should be disaggregated to provide more specific information 
about students’ judgements of the software programs. 
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Methodological 


Generalizability 
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Rigor 
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Evaluation 


Goal 1: Integration 


Technology Impact 


0 


0 


0 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology: 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation: conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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7/99 


-Brenda Curtis; Falls V 
Elementary 

Tiebreaker Elementary, Idaho 
Falls School Diistrict 


Can Students’ Capitalization Skills Improve 
by Using Skills Bank Cornerstone Software 
Exclusively as the Teaching Tool? 



Summary (Report #30) 



Purpose: This action research examined the impact of the Skills Bank 

Cornerstone software on students’ development of capitalization 
skills 



Data sources: 



Two classes of fourth-grade students at two different schools 
provided pre- and post-test data. 



Results: Students improved capitalization skills. 

Authors’ Conclusions: 1) Study was beneficial to both the researchers and the students 

2) Most students improved from pre-test to post-test and seemed 
to enjoy the experience. 3) Pre-test/post- test analysis hampered 
by a ceiling effect from inappropriate version of Cornerstone 
software. 



Reviewer’s Comments: Data are difficult to understand. Lack of a control group 

compromises results. 
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Evaluation Assessment by Applicable lETI Goals 



Gosl 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 1 0 



1 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 



BEST COPY AVAILABLE 



A- 70 



Buck Institute for Education- Appendix A 



1999 


Martin Horesji, Idaho State 


Field-based Technology in Idaho Middle 




University 


School Science Classes: An Evaluation of 






Performance and Attitude Data from Students 






(Dissertation Abstract) 



Summary (Report #31) 

Purpose: This dissertation sought to determine whether students who 

used computer technology had more positive attitudes toward 
science and learned more than students who did not use the 
technology. 

Data sources: Middle school students in four earth science classes taught by 

two teachers. 

Results: The use of computer technology in an existing earth science 

curriculum increases student science learning but has no impact 
on student attitudes toward science. 

Authors’ Conclusions: 1) Study provides empirical data on the impact of computer 

technology on student learning: 2) The introduction of 
computer technology may be subject to a “ceiling effect” where 
the inclusion of additional computer technology into the 
classroom no longer leads to learning gains. 

Reviewer’s Comments: A dissertation abstract, by its very nature, does not provide 

much substantive information, and I have little information 
from which to Judge the quality of the research. Still, the author 
uses a “post-test only” design which, without random 
assignment, does not account for pre-existing differences in 
academic achievement. Findings thus may result from 
differences in the students enrolled in the classes rather than the 
use of technology. The speculations regarding an optimal 
amount of classroom computer technology, and the time line for 
integration of this technology are intriguing and potentially 
important for Idaho and American education. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological 

Rigor 



Generalizability 
to Idaho 
Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Technology Integration 

Technology Impact 0 0 1 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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No Date 


Brian Hardy, Valley Middle 


Will the Use of a Computer Program Improve 




School. Valley Junior-Senior 


Scores on a Test of Location of the 50 States? 




High School, Valley School 
District 


(Research Summary) 



Purpose: 



Data sources: 



Results: 

Conclusions: 



Summary (Report #32) 

For the past three years the teacher has not been satisfied with 
students’ ability to locate the position of American states. This 
action research examined whether student performance would 
improve if he supplemented his teaching with the computer 
program National Inspirer 4.0. 

Forty-six seventh grade students completed pre- and post-tests 
on the location of the 50 American states. 

Students' average score increased from pre-test to post-test 

No conclusions drawn. 



Reviewer’s Comments: Good documentation of the effectiveness of one teacher. 

Without a control group, it is impossible to separate the impact 
of the software from other instructional strategies. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 


Methodological 


Generalizability 


Overall Quality 




Rigor 


to Idaho 
Teachers and 
Students 


of Research/ 
Evaluation 


Goal 1: Integration 


Technology Impact 


1 


0 


1 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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No Date 


Jennifer Hardy, Special 


Will the Integration of the Computer into the 




Education, Valley School 


Math Curriculum Increase a Student's 




District 


Mastery of the Multiplication Facts? 
(Research Summary) 



Summary (Report #33) 

Purpose: The subject of the research is a mentally retarded male who has 

been attempting to master math facts for IV 2 years. The student 
was bored by flash cards and worksheets. The teacher decided to 
integrate the computer into the curriculum to see if it would 
help the student learn math facts. 

Data sources: One special education student completed a pre-test and a post- 

test on his knowledge of multiplication facts 0-9. For two 
months he worked on math facts using either Milts Math or 
Math Blaster software for 15-20 minutes a day for 4-5 days a 
week. 

Results: The student scored 89% on the post-test . 

Conclusions: Integrating a computer into the curriculum will not only help a 

student perform better and master more material, but is also a 
motivating and enjoyable way to learn 

Reviewer’s Comments: Good documentation of the progress of one student. It 

demonstrates that students can learn math facts using software 
programs, but gives no information about whether it is more (or 
less) efficient method compared to traditional means. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological 

Rigor 



Generalizability 
to Idaho 
Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 1 



0 



1 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology: 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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7/99 


Donna Henry. Wendell 


What is the Progress of Fourth Grade 




Elementary School, Wendell 


Students Using the Accelerated Reading 




School District 


Program? 



Summary (Report #34) 



Purpose: This action research tracked the growth of fourth-grade 

students’ reading during the 1998-99 school year. 

Data sources: Twenty-eight fourth-grade students were assessed three times 

during the school year using the STAR. Reading Software. 



Results: Data reported for 9 students show inconsistent growth patterns. 

Authors’ Conclusions: 1) There was an overall improvement in students’ grade 

equivalency scores. 2) STAR was useful as a concrete way to 
evaluate students’ reading level and reading development. 3) 
Further research is needed with larger samples. 



Reviewer’s Comments: Although students averaged an increase of 0.1 Grade 

Equivalency score, actual change scores ranged between -0.9 and 
+1.3. Such variability challenges the conclusion that “there was 
an overall improvement in students’ grade equivalency scores. 
Without a control group, it is impossible to isolate the role the 
software in increasing students’ reading skills. 



Buck Institute for Education « Appendix A 



A-77 



ERIC 



122 



Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological 

Rigor 



Generalizability 
to Idaho 
Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 0 0 



0 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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7/99 


Cindy Hull, Oakwood 


Does Accelerated Reading Improve Reading 




Elementary School, Preston 
Joint School District 


Comprehension at Oakwood Elementary? 



Summary (Report #35) 



Purpose: 



Data sources: 



This action research examined the usefulness of the Accelerated 
Reading software program in teaching typing to 7^^ and 8^^ 
grade students. 

Seven classes of third grade students were assessed at the 
beginning and end of the semester using the STAR Reading 
Software. 



Results: 

Authors’ Conclusions: 
Reviewer’s Comments: 



Every class demonstrated improvement in reading level. 

1) Accelerated Reader appears to help students increase their 
reading comprehension skills. 2) Further research is needed 
with larger samples. 

Although there is a trend toward improvement, the results are 
confusing and would benefit from analysis of treatment 
differences among classes, and performance according to gender 
and ethnicity. Without a control group, it is impossible to 
determine the role the software in increasing students’ reading 
skills. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 


Methodological 

Rigor 


Generalizability 
to Idaho 
Teachers and 
Students 


Overall Quality 
of Research/ 
Evaluation 


Goal 1: Integration 








Technology Impact 


0 


0 


0 



Key to Ratings 





Methodological Rigor 




Generalizability 




Overall Quality 








to Idaho Teachers and 




of Research/ 








Students 




Evaluation 


0 


Serious methodological 


0 


Results can not be generalized 


0 


Untrustworthy; conclusions 




problems invalidate results 




beyond current 
research/evaluation 




should be ignored 


1 


Methodological problems 


1 


Results can be generalized 


1 


Flawed research/evaluation; 




compromise trustworthiness 




with caution 




consider conclusions 




of results 








skeptically 


2 


Acceptable methodology; 


2 


Results widely generalizable 


2 


Competent research/ 




trustworthy results 








evaluation; conclusions 
appear sound 


3 


Excellent methodology; 






3 


Exemplary research/ 




compelling results 








evaluation; conclusions have 
import for Idaho education 
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7/99 


Marne McKnight, Stuart 


Will a software program help students learn 




Elementary School, Shelley 
Joint School District 


addition and subtraction facts? 



Summary (Report #36) 



Purpose: This action research examined the effectiveness of a software 

program (Speed Math) as an instructional tool for learning 
addition and subtraction facts. 



Data sources: Twenty-one third-grade students completed a pre-test and then 

used the software for four minutes a day, three days a week, for 
four weeks. 



Results: 1) There was no difference in pre- and post-test scores for 

addition. 2) Students demonstrated greater subtraction skills at 
the end of the program than they did at the beginning. 

Authors’ Conclusions: 1) Future research should be conducted with larger numbers of 

students and include a control group. 

Reviewer’s Comments: At the time of the pretest, the students in this research already 

knew their addition facts; it would be difficult to show dramatic 
gain scores in addition. They were not as well versed in 
subtraction, and it was possible to show growth. Without a 
control group, it is impossible to know the role the software 
played in their skill development. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 



0 



1 



Key to Ratings 



Methodological Rigor Generalizability Overall Quality 

to Idaho Teachers and of Research/ 

Students Evaluation 



0 Serious methodological 0 

problems invalidate results 



1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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No Date 


Dixie Parton & Ann Reynolds, 


Impact of Accelerated Reader Implementation 




Horizon Elementary School, 
Jerome School District 


on Reading Achievement 



Purpose: 



Data sources: 



Results: 



Conclusions: 



Reviewer’s Comments: 



Summary (Report #37) 

To evaluate the effect of the schoolwide implementation of 
Accelerated Reader Software over a three year period. 

Five hundred students, grades 1-6, attending Horizon 
Elementary School from 1996-1999. 

1) Students* reading achievement scores consistently increased 
over the three year period, although the rate of increase varied by 
grade and year; 2) Library circulation has increased from 
approximately 38,000 to approximately 61,000 books during the 
first two years of the program. 

School Renaissance Program implementation has had a 
significant impact on reading improvements at Horizon 
Elementary School. 

Good documentation of students’ reading achievement. Without 
a control group, it is impossible to separate the impact of the 
software from other instructional strategies or from the normal 
growth that would be expected over time. Without more 
information about the STAR test, it is impossible to know 
whether the competencies it measures are similar to those 
assessed by more commonly used measures reading 
achievement. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 



0 



1 



Key to Ratings 



Methodological Rigor Generalizability Overall Quality 

to Idaho Teachers and of Research/ 

Students Evaluation 



0 Serious methodological 0 

problems invalidate results 



1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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7/99 


Dorrie Prange, Salmon 


Comparing the writing with paper and pencil 




Elementary School, Salmon 
School District 


and Hvper studio 



Summary (Report #38) 



Purpose: This action research compared the number of words written and 

facts described when students use Hyper studio compared to 
paper and pencil. 



Data sources: Twenty first-grade students were formed into teams of two and 

these teams rotated among 4 centers. One center was a computer 
running Hyper studio and one was a table with paper and pencil 
and crayon. 



Results: 1) Students wrote more and included more facts when using 

Hyper studio . 2) Student collaboration was easier when students 
were using the computer than when they were writing with paper 
and pencil 

Authors’ Conclusions: 1) The multimedia environment inspired the T* graders to work 

harder. 2) The study should be replicated with older students 
and a larger sample. 



Reviewer’s Comments: 



Intriguing research compromised by the small number of 
students, and the short duration of the study. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological 

Rigor 



Generalizability 
to Idaho 
Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact I 



0 



I 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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7/99 


Jean Shawver, Kimberly High 


Does the Use of Word Processing Software 




School, Kimberly School 


Increase Test Scores on the Direct Writing 




District 


Assessment Test? 



Summary (Report #39) 

Purpose: This action research examined whether students who completed 

the Idaho Direct Writing Assessment received higher scores than 
those who wrote their answers to the test by hand. 

Data sources: Ninth- and twelfth-grade students completed the Direct Writing 

Assessment. Students were allowed to choose between using a 
computer and writing by hand. Thirty-three students (21 boys 
& 12 girls) used the computer while 67 students (39 boys and 
28 girls) elected to write by hand. Data was collected at the 
school and scored by the state. 

Results: 1) A larger proportion of students who used a computer to 

complete the Direct Writing Assessment '^passed above grade 
lever than did students who wrote by hand. 2) A larger 
proportion of students who wrote by hand ''passed at grade level” 
than did students using a computer. 3) The failure rate of 
students using the computer and writing by band was the same. 

Authors’ Conclusions: 1) If a student believes that the computer will aid him or her to 

produce a better product then it will. 2) Word processing gives 
students a greater amount of time to process and evaluate ideas. 
3) Further research with a larger, more balanced sample is 
needed. 

Reviewer’s Comments: Given the design of the research, it is impossible to disentangle: 

1) Are students who chose to use the computer better writers to 
begin with (e.g., they would achieve equivalent results with 
paper and pencil)? and 2) Does using a computer to complete 
the Direct Writing Assessment make one a better writer? 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological 

Rigor 



Generalizability 
to Idaho 
Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 0 0 0 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology: 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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No Date 


Walter D. Thueson, Shoshone 


What is the Impact of Computerized Reading 




School District 


Tests and Scoring on Student Reading 
Performance? (Research Summary) 



Summary (Report #40) 



Purpose: 



Data sources: 



Results: 



Conclusions: 



This evaluation was undertaken to assess the effectiveness of the 
Accelerated Reader Program. 

Students in grades 8-12 completed the STAR reading assessment 
in Fall 1997, Spring 1998, and Fall 1998. 

Students showed an overall 1.05 grade placement increase from 
Fall 1997 to Fall 1998; Students in grades 11 and 12 showed an 
average increase of 2.5 grade levels. Students receiving the 
Waterford program out-performed the control group by a 
significant margin (p < .06) 

The Waterford program has been implemented in all 34 district 
elementary schools. 



Reviewer s Comments: Although details are sketchy, the study appears to be 

satisfactorily designed. The use of an outcome measure supplied 
by the software manufacturer raises some questions. It is 
possible the students in the Waterford program scored higher 
than the control students because the outcome measure more 
closely matched what they had been taught. A more convincing 
demonstration would employ a “neutral” assessment 
instrument. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 1 1 



1 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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Theda Torgerson, Anne 
Heyrehend, & Dennis Parker, 
Groveland Elementary School, 
Blackfoot School District 



Assessing the Effectiveness of the Skills Bank 
Lansuage Computer Program 



Purpose: 



Data sources: 



Results: 



Authors’ Conclusions: 



Summary (Report #4 1) 

This action research evaluated the effectiveness of the Skills 
Bank Language computer program and the training and 
technical support teachers received. 

Third-grade students were tested before and after they used the 
Skills Bank software. Teachers completed questionnaires at the 
beginning and at the conclusion of an inservice program. 

1) Students demonstrated gains in capitalization, punctuation 
and language usage. 2) Implementation was compromised by lack 
of administrative support and network computer problems. 

1) Students’ language skills improved, although without a 
control group it is impossible to distinguish the impact of the 
Skills Bank software. 2) Increased administrative support and 
the designation of a control is necessary before further research 
is conducted. 



Reviewer’s Comments: 



Research report lacks essential data about the number and 
nature of the students involved in the program and the amount 
of time they used the software. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 0 0 0 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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7/99 


Carolyn Tufts, Georgetown 


Do students learn addition facts faster if they 




Elementary School, Bear Lake 
County School District 


use a computer? 



Summary (Report #42) 



Purpose: This action research examined whether students who used a 

computer to learn basic addition facts were able to solve 
problems more quickly and more accurately than students who 
only used paper and pencil. 

Data sources: Twenty-seven second-grade students completed pre- and post- 

tests of addition facts. 

Results: 1) 25 out of 27 students completed more problems on the post-test 

than on the pretest; 2) 24 out of the 27 students answered more 
questions correctly on the post-test than they did on the pretest; 

3) The accuracy of the responses of 11 out of 27 students 
decreased. 



Authors’ Conclusions: 1) the number of problems completed and the number done 

correctly was directly related to the amount of time students 
spent using the computer. 

Reviewer’s Comments: It appears that both the experimental and the contrast group 

used the computer to master addition facts. If this is indeed the 
case, then the author’s conclusions are erroneous. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 0 0 0 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology: 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy: conclusions 
should be ignored 

1 Flawed research/evaluation: 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation: conclusions 
appear sound 

3 Exemplary research/ 
evaluation: conclusions have 
import for Idaho education 
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No Date 


John Davis, Doug Amos, 


An Evaluation of Traditional and Computer- 




University of Idaho-Moscow 


Enhanced Approaches to Teaching 
Mathematics and Language Arts (Research 
Summary) 



Purpose: 



Data sources: 



Results: 



Conclusions: 



Summary (Report #43) 

Is there a statistically significant difference between a participant 
group using microcomputers as enhancement aids (treatment) 
and a participant group using traditional enhancement aids 
(control)? 

Fifty-three elementary teachers participated in two treatment 
groups. The first group received traditional instruction and 
microcomputer enhancements for 30 minutes a day. The second 
group used worksheet and workbook activities for 30 minutes 
each day following traditional instruction. Specific sub-tests 
from the Iowa Test of Basic Skills were used as pre and post 
measures. 

The group with computer-enhanced instruction showed a 
moderately higher change score than the traditional group. 

The use of computers assists student learning in mathematics 
and language arts when implemented in conditions similar to 
those used. 



Reviewer’s Comments: This research summary doesn’t indicate whether there was a 

statistically significant difference between the groups, nor is 
information presented about the initial comparability of the 
groups. The lack of information about the length of treatment 
or the type of computer-enhanced activities makes it difficult to 
generalize from the study’s results. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological 

Rigor 



Generalizability 
to Idaho 
Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Integration 

Technology Impact 0 0 0 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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10/18/99 Judith Doerann, Independent PDS: Professional Development Systems 
Evaluation Consultant Evaluation Report (DRAFT Technology 

Literacy Challenge Grant Evaluation) 





Summary (Report #44) 


Purpose: 


This evaluation report examined the effectiveness of the 
Professional Development System (which combines a technology 
training model and one-omone support) in helping teachers to 
integrate technology into the study of Idaho history. 


Data sources: 


Project implementation data was provided by the Project 
Facilitators: teachers provided self-reported data on their 
technology competencies and their technology attitudes, beliefs, 
and use and had the opportunity to complete the Teacher 
Technology Competency Test developed by Boise State 
University. 


Results: 


1) The Project Facilitator was effective in helping participants 
increase their knowledge of and comfort with technology 
integration; 2) Four participants who failed the Teacher 
Technology Competency Test before the training passed it at the 
conclusion; 3) Teachers increased their use of multimedia and 
word processing and used more project-based, cooperative 
learning activities in their classrooms; 4) The Coeur d'Alene 
School District will incorporate the PDS project into their District 
Technology Plan, 


Authors’ Conclusions: 


Limitations in the data make it difficult to form firm, focused 
conclusions derived from the evaluation results. Still, the 
evidence provided by the participants’ use of technology, their 
passing the Teacher Technology Competency Test, and their 
increased knowledge of integration in multimedia, word 
processing, and graphics aligned with the goals and activities of 
the PDS project and most likely are evidence of project impact. 


Reviewer's Comments: 


The conclusions of this evaluation go far beyond the evidence 
presented - a situation more or less acknowledged in the 
document. I urge the project team to consider whether such 
evaluations provide useful data for program improvement and 
are a good use of limited resources. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological 

Rigor 



Generalizability 
to Idaho 
Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Technology Integration 

Technology Training 0 0 0 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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Summer 


Judith Doerann. Independent 


Excel in Mathematics Evaluation Report 


1999 


Evaluation Consultant 


(Technology Literacy Challenge Grant 
Evaluation) 



Summary (Report #45) 

Purpose: This evaluation report examined the effectiveness of Project 

EXCEL in increasing students’ understanding of spreadsheets, 
integrating appropriate technology into curriculum and 
instruction, increasing family involvement in mathematics 
education, and providing equal access to available technology. 

Data sources: Project implementation data was provided by the Project 

Facilitators; Student achievement data was provided by twenty- 
two classroom teachers; Teacher technology competency was 
assessed by pre- and post-administration of the Teacher 
Technology Competency Test developed by Boise State 
University; Teacher use of and attitudes toward technology was 
assessed by teacher self-report instruments; Family information 
was provided by a pre- and post-questionnaire. 

Results: 1) Student scores showed an increase in knowledge of 

spreadsheets and graphing; 2) Teachers increased technology 
knowledge and integration; 3) The attitude of most parents 
towards mathematics homework improved; parents wanted the 
project extended; parents did not, however, become more involved 
in students mathematics homework; 4) The project provided all 
students with access to technology. 

Authors’ Conclusions: Project Excel in Mathematics achieved most of its goals in nearly 

all of the participating classrooms. Teacher knowledge of 
technology and its integration into education increased. 

Reviewer’s Comments: This evaluation documented what happened in a single project; 

it was not designed to contrast computer-enhanced and 
traditional instructional approaches. Consequently, it does not 
provide evidence of educational value added by the integration 
of computer technology - although one could argue that it is 
impossible to teach about spreadsheets without computers, and 
therefore, computer technology adds value by allowing the 
teacher to include spreadsheets in the curriculum. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Technology Integration 

Technology Training 10 1 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 

3 Excellent methodology; 
compelling results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy; conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation; conclusions 
appear sound 

3 Exemplary research/ 
evaluation; conclusions have 
import for Idaho education 
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Summer 


Judith Doerann, Independent 


Project TROUT: Teachers Reveal. Organize 


1999 


Evaluation Consultant 


and Understand Technology Evaluation 
Report (Technology Literacy Challenge Grant 
Evaluation) 



Summary (Report #46) 

Purpose: This evaluation report examined the effectiveness of impact of 

Project TROUT in helping teachers to master the teacher 
competencies in the International Society of Technology 
Education Standards and to integrate technology in classroom 
instruction and for personal use. 

Data sources: Information provided by project facilitator; pre- and post- 

administration of the Teacher Technology Competency Test 
(developed by Boise State University); questionnaires developed 
by the evaluator; and topical quizzes to ten participating 
teachers. 



Results: 1 ) Fifty percent of the teachers passed the Teacher Technology 

Competency Test; 2) Pre measures were not available until after 
the project had already started, thereby making it impossible to 
evaluate change in participant s learning; 3) Instructional model 
was powerful in increasing teachers' confidence and technological 
competence; 4) At the conclusion of the project, teachers used 
technology more and for longer periods of time. 



Authors’ Conclusions: The project was generally effective in reaching project goals of 

professional development for teachers in technology integration, 
and improvement in personal and professional use of 
technology. 



Reviewer s Comments: 



It appears that considerable effort was spent evaluating the 
impact of a professional development program on 10 teachers - 
despite the lack of assessment procedures to document changes 



in teachers’ technology skill levels as a result of program 
participation. I suggest that the project director and evaluator 
consider carefully whether such small-scale, methodologically 
flawed evaluations provide data useful for program 
improvement or validation. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological Generalizability 
Rigor to Idaho 

Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Technology Integration 

Technology Training 0 0 1 



Key to Ratings 





Methodological Rigor 




Generalizability 




Overall Quality 








to Idaho Teachers and 




of Research/ 








Students 




Evaluation 


0 


Serious methodological 


0 


Results can not be generalized 


0 


Untrustworthy; conclusions 




problems invalidate results 




beyond current 
research/evaluation 




should be ignored 


1 


Methodological problems 


1 


Results can be generalized 


1 


Flawed research/evaluation; 




compromise trustworthiness 




with caution 




consider conclusions 




of results 








skeptically 


2 


Acceptable methodology; 


2 


Results widely generalizable 


2 


Competent research/ 




trustworthy results 








evaluation; conclusions 
appear sound 


3 


Excellent methodology; 






3 


Exemplary research/ 




compelling results 








evaluation; conclusions have 
import for Idaho education 
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10/18/99 



Judith Doerann, Independent 
Evaluation Consultant 



PROJECTS Focus and Reaching Academic 
Excellence through Staff Development 
Evaluation Report (DRAFT Technology 
Literacy Challenge Grant Evaluation) 



Summary (Report #47) 



Purpose: This evaluation report examined the effectiveness of two 

projects in advancing staff toward meeting the ICTL Standards 
in Technology Integration. 

Data sources: Teachers provided self-reported data on their technology 

competencies and their technology attitudes, beliefs, and use 
and had the opportunity to complete the Teacher Technology 
Competency Test developed by Boise State University. 

Results: 1) On average, teachers were more successful on the post-test 

than the initial test of the Teacher Technology Competency 
Assessment; 2) Teachers used technology more frequently in the 
classroom at the conclusion of the training; 3) Teachers were 
eager to assist and collaborate with other staff and to expand 
their knowledge and skills. 



Authors’ Conclusions: Although the intended pre-measures were not available at the 

start of the project, the evaluation results suggest that the 
project was generally effective in reaching project goals for 
professional development for teachers in technology integration. 



Reviewer’s Comments: The conclusions of this evaluation go far beyond the evidence 

presented - a situation more or less acknowledged in the 
document. I urge the project team to consider whether such 
evaluations provide useful data for program improvement and 
are a good use of limited resources. 
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Evaluation Assessment by Applicable lETI Goals 



Goal 



Methodological 

Rigor 



Generalizability 
to Idaho 
Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



Goal 1: Technology Integration 

Technology Training 0 0 0 



Key to Ratings 



Methodological Rigor Generalizability 

to Idaho Teachers and 
Students 



Overall Quality 
of Research/ 
Evaluation 



0 Serious methodological 
problems invalidate results 

1 Methodological problems 
compromise trustworthiness 
of results 

2 Acceptable methodology; 
trustworthy results 



0 Results can not be generalized 
beyond current 
research/evaluation 

1 Results can be generalized 
with caution 

2 Results widely generalizable 



0 Untrustworthy: conclusions 
should be ignored 

1 Flawed research/evaluation; 
consider conclusions 
skeptically 

2 Competent research/ 
evaluation: conclusions 
appear sound 



3 Excellent methodology: 
compelling results 



3 Exemplary research/ 

evaluation: conclusions have 
import for Idaho education 



A- 104 



Buck Institute for Education- Appendix A 



Author Form 



http://ericir.syr.edu/ithome/datab...se_Forms/2001-Mar-2 1-19:21: 15.html 




U.S. Department of Education 

Office of Educatonal Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE 
(Specific Document) 






I. DOCUMENT IDENTIFICATION: 



Title: 


An Analysis of the Progress of the Idaho Educational Technology Initiative in Meeting 
Goals Established by the Idaho Council for Technology in Learning 


Authors: 


John R. Mergendoller, Kevin Moriarty 


Corporate Source: 


Buck Institute for Education Publication Date: 


12/99 



II. REPRODUCTION RELEASE: 

In order to disseminate as widely as possible timely and significant materials of interest to the educational community, 
documents announces in the monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made 
available to users in microfiche, reporduced paper copy, and electronic media, and sold through the ERIC Document 
Reproduction Service (EDRS). Credit is given to the source of each document, and, if reproduction release is granted, one of 
the following notices is affixed to the document. 

If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three 
options below and sign at the bottom of the page. 



■ 


The sample sticker shown below will be affixed to 
all Level 1 documents 


The sample sticker shown below will be affixed to 
all Level 2A documents 


The sample sticker shown below will be afiixed to 
all Level 2B documents 


PERMISSION TO REPRODUCE AND 
^ DISSEMINATE THIS MATERIAL HAS BEEN 
I GRANTED BY 


PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE, AND IN ELECTRONIC MEDIA 
FOR ERIC COLLECTION SUBSCRIBERS 
ONLY,HHAS BEEN GRANTED BY 


PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE ONLY HAS BEEN GRANTED 
BY 


! TO THE EDUCATIONAL RESOURCES 

1 INFORMATION CENTER (ERIC) 


TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 


TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 


Level 1 


Level 2A 


Level 2B 




o 


O 


Check here for Level 1 release, permitting 
i reproduction and dissemination in microfiche or 
^ other ERIC archival media (e.g., electronic) and 
paper copy. 


Check here for Level 2A release, permitting 
reproduction and dissemination in microfiche and 
in electronic media for ERIC archival collection 
subscribers only. 


Check here for Level 2B release, permitting 
reproduction and dissemination in microfiche only 


1 Documents will be processed as indicated provided reproduction quality permits. 

1 If permission to reproduce is granted, but no box is checked, documents will be processed at Level 1 . 



I hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and 
i disseminate this document as indicated above. Reproducation from the ERIC microfiche or electronic media by persons 
I other than ERIC employees and its system contractors requires permission from the copyright holder. Exception is made 
1 for non-profit reproduction by libraries and other service agencies to satisfy information needs of educators in response to 
discrete inquiries. (Typing your name below is the legal equivalent of signing a paper document.) 


Name (Signature): 


John R. Mergendoller 


Position/Title: 


Executive Director 


Organization/Address : 


Buck Institute for Education 18 Commercial Blvd. Novato, CA 94949 


Telephone: 


415.883.0122 


FAX: 


415.883.0260 | 


E-MAIL Address: 


john@bie.org 


Date: 


3721/01 






3/22/01 8:34 AM 




Authof|Form 



http://ericir.syr.edu/ithome/datab...se_Forms/2001 -Feb-26-1 3:27:44.html 




U.S. Department oflEducation 

Office of Educatonal Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 

REPRODUCTION RELEAM 
(Specific Document) 






I. DOCUMENT IDENTIFICATION: 



Title: 


^ PrOGMSS of the IDAHO ^ | 

TECHNOLOGY INITIATIVE IN MEETING GOALS ESTABLISHED BY THE 
IDAHO COUNCIL FOR TECHNOLOGY IN LEARNING 


Authors: 


JOHN R. MERGANDOLLER Ph.D. AITO ^vin PRINCETON 

FELLOW 


Corporate Source: 


OFFICE OF THE GOVERNOR OF THE STATE OF 
IDAHO AND BUCK INSTITUTE FOR 
EDCUATION 


Publication Date: 


December, 1999 



II. REPRODUCTION RELEASE: 

In order to disseminate as widely as possible timely and significant materials of interest to the educational community, 
documents announces in the monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made 
available to users in microfiche, reporduced paper copy, and electronic media, and sold through the ERIC Document 
Reproduction Service (EDRS). Credit is given to the source of each document, and, if reproduction release is granted, one of 
the following notices is affixed to the document. 

If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three 
options below and sign at the bottom of the page. 



The sample sticker shown below will be affixed tq 
all Level 1 documents 


The sample sticker shown below will be affixed to 
all Level 2A documents 


The sample sticker shown below will be affixed tol 
all Level 2B documents 


PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS BEEN 
GRANTED BY 


PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE, AND IN ELECTRONIC MEDIA 
FOR ERIC COLLECTION SUBSCRIBERS 
ONLY,HHAS BEEN GRANTED BY 


PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE ONLY HAS BEEN GRANTED 
BY 






TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 


TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 


TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 


Level 1 


Level 2A 


Level 2B 




o 


o 




Check here for Level 1 release, permitting 
1 reproduction and dissemination in microfiche or 
^ other ERIC archival media (e.g., electronic) and 
paper copy. 


Check here for Level 2A release, permitting 
reproduction and dissemination in microfiche and 
in electronic media for ERIC archival collection 
subscribers only. 


Check here for Level 2B release, permitting 
reproduction and dissemination in microfiche only 



Documents will be processed as indicated provided reproduction quality permits. 

If permission to reproduce is granted, but no box is checked, documents will be processed at Level 1 . 



\ i hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and 
! disseminate this document as indicated above. Reproducation from the ERIC microfiche or electronic media by persons 
\ other than ERIC employees and its system contractors requires permission from the copyright holder. Exception is made 
for non-profit reproduction by libraries and other service agencies to satisfy information needs of educators in response to 
\ discrete inquiries. (Typing your name below is the legal equivalent of signing a paper document.) 


^ Name (Signature): 


Robert C. West, Ph.D. 


Position/Title: | 


Chief Deputy Superintendent 


^ Organization/Address: 


Idaho Department of Education P. 0. Box 83720 Boise, ID 83720-0027 


1 Telephone: 


(208)332-6810 


FAX: 


(208) 334-2228 


1 E-MAIL Adidress: 


bwest@sde.state.id.us | 


Date: | 


February 26, 2001 




2/26/01 2:23 PM 




